backup exec 12.5 stall - e0008821

So every once in a while one of my policy based disk to disk to tape backup jobs will just hang or stall forever. It's really nifty since it will go well beyond the auto-cancellation period - you know the ' stop job if it takes longer than x hours' checkbox. The first few times it happened I wound up having to restart the whole backup server to get the job to cancel since no amount of effort on the front end GUI would fix it. After reboot it would show the failed job with a generic error code of e0008821. I even tried restarting the server that the job status claimed it was working on at the time of the hang. After a few more tries it occurred to me that the job status was lying to me and that it may actually be trying to communicate with the next server on the backup list. So to the command prompt I went. The results of netstat -aon | more

show that the server is currently communicating with port 10000 on another server. The catch is that while the job status said it was working on server A, the netstat showed it was only talking to server B at the time. So I went to server B and restarted the backup Agent and my stalled job suddenly started proceeding again. Of course, this isn't a great long term fix but it's not unusual for backup agents to need a good kick start from time to time. It would be nice if it was better at auto-recovery...

Step by step creating a Shared Scope in Search Server 2008 and making it show up in the dropdown

So you've got your Search Server 2008 installed and working but you now want to get creative and add scopes to limit results. Seems reasonable enough. If you just try creating a scope from one of your site collections you may find that it doesn't give you any results when you try to use it and you get a nifty error message as well like "scope in your query does not exist".

So instead, you need to create a Shared scope and then make a copy of that scope in your site collection for it to work.

1. First create the shared scope from the Central Admin. Note the mode=ssp at the end!
http://YourServerName:yourportnumber/_layouts/viewscopesssp.aspx?mode=ssp
Click on New Scope, fill in the options and OK out of it.


2. You'll now see a that says Empty - Add Rules. Click on that. Here's where you'll set the limit on what sources you want results returned for. In my case, I chose content and limited my scope to only the sites my sharepoint server.

The finished Shared Scope should look like this:

3. You may or may not see a message saying that it will update in X minutes. If you see that, wait X minutes before continuing.

4. Now that you have your Shared Scope, go into one of your sites that you want to add this drop down to. (Yes, this will need to be done for every site that you want to add this to). Click Site Actions, Site Settings and under Site Collection Administration you should see "Search Scopes". If you don't, then log in as someone with more rights. Once you get into it, you should see a screen that by default has "Search Dropdown", "Advanced", and "Unused". At this point you'd love to be able to just drag it over and be done but alas, we're not there yet.
5. Click on the drop down arrow next to your Shared Scope and choose "Make copy"

6. Then click on the drop down arrow next to your new copy and click on Edit Properties and Rules.


7. You'll notice that unlike the Shared Scope, you can actually check the boxes for "Search Dropdown" and "Advanced Search". Rename the scope to what you want to display, check the boxes, and then click OK.


8. Now you should see your new copied scope show up in the right categories. You may or may not see a message saying that it will update in X minutes. If you see that, wait X minutes before continuing.

9. Now if you go into your site and refresh the page, you should see your new drop down option in the search box. (If you've created a "SearchCenter" page, remember to edit the WebPart first to actually show the drop down box.) Test it out to make sure it's working.

10. Now rinse, lather, and repeat the same for all the sites that you want this scope added to.

Credit where credit's due:
Aboo Bolaky's page - http://www.bolaky.net/?tag=/msse Good clue for finding the mode=ssp starting point.

And a thanks to the countless blogs out there that I used to set up search server 2008 in the first place! (It only took me 5 tries, thank heaven for virtual machines and rollback!)

Search Server 2008, ISA 2006, AAM, and why All Sites wasn't showing up externally

So I've had my WSS 3.0 SP2 site running for quite some time now and had all the publishing in place for ISA Server 2006 so my guys could get access to it remotely. I finally decided to try out Search Server 2008 Express which took a few tries to get installed correctly. (Thank heaven for virtual machines and snapshot rollback!). So now all my sites have an extra drop down under Search for "All Sites" and I now have a search center site collection for users to hit directly as well. Problem is, that it wasn't showing up externally and the search center site completely barfed for my external users.

At this point I was pretty sure AAM (Alternate Access Mapping) was involved but I just wasn't sure why Search was so special. It took a couple of hours but I figured it out. There's a ton of good websites out there that show you how to set up sharepoint and ISA server so I'm just going to focus on what I missed and skip the basics.

Mistakes:
1. I had never 'Extended' the Web Application. I only had a Default Zone Web Application.
2. My ISA publishing rule was using the Intranet Zone name for the server. Which means that all external users were being seen by the Sharepoint server as using the Intranet Zone even thought they were using the Internet address. (i.e. They typed in sharepoint.yourdomain.com and the ISA server sent on sharepoint.intranet.yourdomain.com). This was never a problem before search but it became one afterwards.


Warning: I am by no means an expert on Sharepoint, I just play one at work. The following comprises the most critical things I had to change to get mine to work. Always back up your systems before making changes to production environments.

So to fix it, I first went into Central Administration, Application Management and then "Create or Extend Web Application"



Then "Extend an Existing Web Application"


Then click the drop down for Web Application and make sure the right website is selected. Then fill in the port (in almost all cases its 80) and the Host Header (REQUIRED - since we'll be stacking onto the same Port as other Sites on the same server). I use Kerberos but if you haven't set up your SPNs then choose something else. Make sure to choose the Intranet zone (or whatever yours corresponds to)

Now either Restart IIS manually, or iisreset /noforce, or if you're bored reboot the whole server.
When it comes back you should be done. Test it from an external location.

For reference, here's my AAM settings (modified of course.) And I wouldn't worry if yours don't match exactly, sharepoint is just weird that way.



Note: If you had anything published on the original Default site, you'll need to duplicate it into the new Host Header site. i.e. /images or other static content, etc using the IIS Manager. It is, for all intents and purposes, a completely different site.

Useful References:
Microsoft reference - at the bottom talks about Extending Web Applications
http://technet.microsoft.com/en-us/library/cc268368.aspx

Tutorial on publishing sharepoint through ISA
http://www.isaserver.org/tutorials/How-to-Publish-Microsoft-Sharepoint-Service-ISA-Server-2006.html

AAM
http://technet.microsoft.com/en-us/library/cc288609.aspx

More on extending
http://sharepointnotes.wordpress.com/2008/03/02/sharepoint-extranet-solutions-with-isa-server-2006-part-2-creating-the-extranet-web-application/

And more AAM
http://blogs.catapultsystems.com/matthew/archive/2008/05/19/search-results-and-aam.aspx

Why you can't find Hash Publication for BranchCache or Lanman Server under administrative templates.

If, like me, you forgot to update your Central Policy store when you upgraded your AD to 2008 R2, then these won't show up at all.

First confirm that you are using a Central Policy store by opening up any group policy in Group Policy Management and look for the highlighted text.


Once confirmed, now go to \\FQDN\SYSVOL\FQDN\policies\PolicyDefinitions (replace FQDN) and look at the dates. Now compare those with c:\windows\PolicyDefinitions on one of your 2008 R2 AD controllers. If the 2008 R2 has newer files, copy all the contents of that PolicyDefinitions folder to the \\FQDN\SYSVOL\FQDN\policies\PolicyDefinitions, replacing all that's there currently.

Now if you close Group Policy Management, reopen it, and then go back into a policy you'll see new entries including the elusive "Lanman Server" which contains the "Hash Publication for BranchCache" value that you're looking for.

The dreaded e00081d9 The Backup Exec job engine system service is not responding error

It's odd that every time I get backup exec 12.5 SP2 to a nice stable point something inevitably goes wrong. I suddenly started getting this error when the other half of my two backup jobs would run. I tried deleting/re-creating the jobs from policy, liveupdate, beutility- rebuild/repair/etc the db. Each time the backup job engine would puke and fail. Finally I gave up, made a copy of all the mdf and ldf files in the backup exec/data folder and uninstalled the program. And wiped out the whole backup exec folder for good measure. Then I re-installed it, liveupdated it, and then stopped the backup exec and sql services so I could re-swap out the mdf/ldf files. This restored all my jobs/policies/preferences/etc so I wouldn't have to start from scratch. But I still ran into the problem so I ran a manual backup job and it worked so I went ahead and deleted the whole Policy and selection list and made a new one from scratch.

After sleeping on it, it occurred to me that I had made one other change to one of the servers that gets backed up. This other server had died this week and since I only cared about the static files on it I had done a full rebuild of the server including upgrading the OS from 2k3 to 2k8 server. I kept the same name, folder names, etc and installed the beremote agent on it. In retrospect, I think after doing that I should have wiped out the .idr file that automatically gets generated at C:\Program Files\Symantec\Backup Exec\idr\Data. The more I think about it, the more likely it is that by changing the whole OS/setup of a previously snapped IDR box, I had confused the job engine to the point of failure. Of course, for now it's only a theory.

SQL 2008 Transactional Replication and initializing from a backup file

Now there's a fun process to go through, especially if like me you don't know jack about T-SQL. Let's face it, I currently know more words in Mandarin than I know commands in T-SQL; which isn't a lot. Having already done several tests with the automatic method of setting up Transactional Replication (where it does all the initial synch work for you and you just sit back and watch) I had assumed that Initializing from a backup would be a breeze. Famous last words.

There are some articles out there on the web but I found that most of them either assumed you knew more or just left out minor details. If, like me you're trying to set up replication of a huge database over a bandwidth limited connection or if you have some other reason that the initial setup has to be done from a backup file, then here's the walkthrough. By the way, I've only done the one-way Transactional Replication as in my situation this is just a failover site and will not need to send changes back to the original server.

Steps (order is very important)

1. Set up the Distributer – database and share – one time setup
2. Set up Publisher on source server - Don’t use either checkbox for snapshot.
3. Enable the flag under the Publication properties to allow “initialize from backup”. (Right-click on the publication, properties)
4. Disable the distribution cleanup agents. (Under SQL agent jobs)
5. Make a Full backup of the database. Keep a local copy as you'll need it later.
6. Copy database to other site. (Over the network, courier pigeon, magic, whatever)
7. Restore database with the same name
8. Create pull subscription on the destination server. (see scripts provided below)
9. Check status – Replication monitor -> drill down to publisher (add if needed), “View details on the subscription on the right to get a status report (3rd tab in window that pops up)
10. After it’s done synching up, turn off the “initialize from backup” flag or else the cache it keeps will never shrink. And re-enable the distribution cleanup agents.
11. TEST IT. Check the tables after synchronization and then check again after new transactions have been sent. (Wait a few minutes after each replication interval to give it time to catch up).

The reason that order is so important is that after you configure the publisher to enable the "Initialize from backup" and stop the cleanup jobs, it starts keeping a full record of all transactions that have occured since then. (Yes, the DB could grow a lot depending on how long this takes). The backup has a special value in it called an LSN number. This value tells the server to only send transactions that occurred after the backup was made.
Reference:
http://www.replicationanswers.com/NoSyncOn2005.asp

TIP 1: If you try to use a backup that was created before the publication was set up, it will fail.
TIP 2: If you get the Msg 21397 error mentioned in the link above, then you probably forgot to stop the Distribution cleanup agents and the server has thrown out some of the transactions that have occurred since the LSN (backup).

Step 10 is necessary because the distributor will keep waiting around for another subscriber and in the meanwhile your ms_replcommands table will continue to grow.
http://ilkirk.com/archive/2009/01/a-replication-fun-fact/

Now we'll move on to the actual subscription scripts that you will run around step 8. (I'll assume that you set up the Publication through the GUI keeping in mind not to create a snapshot). If you don't know how to create the publication, see the help file or http://msdn.microsoft.com/en-us/library/ms151160.aspx

On the Publishing server we're going to use sp_addsubscription to define the initial subscription, then we'll run sp_addpullsubscription and sp_addpullsubscription_agent on the Subscriber machine. I created the subscriber scripts by using the GUI and choosing the export to script option at the end instead of executing the change. Then I modified the subscriber scripts and that's how I recommend that you set them up. My generalized scripts below should just be used as a guide.

Script 1: (Yes, the exec line is really long. Run on publisher)

-----BEGIN: Script to be run at Publisher 'Publishing_SQLServerName'------------
------- backupdevicename has to be located on the Publisher machine -------

use [Your_DB_Name]
exec sp_addsubscription @publication = N'Your_DB_Name_PUB', @subscriber = N'Subscribing_SQLServerName', @destination_db = N'Your_DB_Name', @sync_type = N'initialize with backup', @backupdevicetype = 'disk', @backupdevicename = 'e:\BACKUP\Your_DB_Name090209.bak', @subscription_type = N'pull', @update_mode = N'read only'
GO
-------END: Script to be run at Publisher 'Publishing_SQLServerName'-------------

Replace Your_DB_Name, Your_DB_Name_PUB, Subscribing_SQLServerName, and the location of the backup with appropriate values.

Script 2: (run at the subscriber sql server)

-----BEGIN: Script to be run at Subscriber 'Subscribing_SQLServerName'-----------------
use [Your_DB_Name]
exec sp_addpullsubscription @publisher = N'Publishing_SQLServerName', @publication = N'Your_DB_Name_PUB', @publisher_db = N'Your_DB_Name', @independent_agent = N'True', @subscription_type = N'pull', @description = N'', @update_mode = N'read only', @immediate_sync = 0
-----END: Script to be run at Subscriber 'Subscribing_SQLServerName'-----------------

Same drill as before with changing out the placeholders with actual names.

Script 3: (still on the subscriber). Now we're going to set up the agents which will handle the data pulls for us. (once again, a really long exec command)

-----BEGIN: Script to be run at Subscriber 'Subscribing_SQLServerName'-----------------
exec sp_addpullsubscription_agent @publisher = N'Publishing_SQLServerName', @publisher_db = N'Your_DB_Name', @publication = N'Your_DB_Name_PUB', @distributor = N'Publishing_SQLServerName', @distributor_security_mode = 0, @distributor_login = N'dist_login_acct', @distributor_password = N'dist_password', @enabled_for_syncmgr = N'False', @frequency_type = 64, @frequency_interval = 0, @frequency_relative_interval = 0, @frequency_recurrence_factor = 0, @frequency_subday = 0, @frequency_subday_interval = 0, @active_start_time_of_day = 0, @active_end_time_of_day = 235959, @active_start_date = 20090902, @active_end_date = 99991231, @alt_snapshot_folder = N'', @working_directory = N'', @use_ftp = N'False', @job_login = N'domain\username', @job_password =N'user_password', @publication_type = 0
GO
-----END: Script to be run at Subscriber 'Subscribing_SQLServerName'-----------------

If you went through the GUI you will have seen where it prompted you for user accounts. The windows account is needed to access the Distribution share on the publisher that houses the snapshots(if we were using them) and updates. Fortunately both machines were on the same domain so that was easy for me. For the distributor I created a SQL login on both servers with the same username/password and granted that user rights on the publishing server. This account is used by the agents on the subscriber to check the distribution database on the publisher.

Provided you didn't get any errors when you ran those scripts, you'll want to start monitoring the replication now. Right click on the Subscription and View Synchronization Status.


Now right-click in SQL Management Studio on Replication and Launch the Replication Monitor. Drill down to the publisher (add if it need be) and then drill down to the publication. Right click on the right panel and View Details. The window that pops up is really useful to see how replication is going.

What's great about that window is that you can watch the # of pending transactions that are waiting to be synch'd. Initially this will be a very big number until it catches up. At this point the only thing left are steps 10 and 11. Turn off the flag on the publisher for Initializing from backup and actually go into the database and replica to see if data is being transferred properly. (taking into account replication intervals, etc)

TIP 3: If you get this error: The distribution agent failed to create temporary files in C:\Program Files\Microsoft SQL Server\100\COM directory. System returned errorcode 5.
Then you need to grant the user that the Distribution Agent is running as Write access to that directory. http://support.microsoft.com/default.aspx/kb/956032

TIP 4: You may start getting errors in your DB related to "Length of LOB data". This occurs because by default it only supports chunks up to 65535. Go into SQL management Studio, right click on the server and choose properties. Set the Max Text Replication Size to something higher. Or do what I did and use the max value of 2147483647.

TIP 5: I found a good reference book that just focuses on SQL replication called "Pro SQL Server 2008 Replication" (ISBN13: 9781430218074). It explains in detail the mechanisms behind replication and covers all types of replication for SQL and how to choose the one that's right for you.

backup exec remote agent stuck in starting state

Server: Windows 2003 SP2
Symantec Backup Exec 12.5 SP2

So after a recent security patch, one of my server's started having backup problems. Then I noticed that the Backup Exec Remote Agent was stuck in 'Starting' under the services panel. Tried killing beremote, rebooting, reinstalling, etc to no avail. So I tried the internet and found a retarted solution that somehow works. Apparently one of the outlook express patches automatically changes the default mail program to outlook express. So you have to go into internet options and set it to Outlook. Even if you don't have outlook installed. Even though this doesn't make a darn bit of sense. Then try starting the service after making that change.

http://www.symantec.com/connect/forums/backup-exec-remote-agent-stuck-quotstartingquot-state

XP remote desktop problem connecting to Windows 7

Before you ask, it's not the Network Level Auth problem.
And no, the computer is on and in the same domain.
And both computers are in the same subnet.
But all the DHCP based XP clients can RDP fine to the Windows 7 boxes. Hmm.

The problem I ran into is that some of the legacy XP clients had static IP addresses set but no "DNS suffix for this connection". So I added it in and voila it stated working.


I didn't really think it'd be that either but I tested it out on 2 more machines with the same results. Apparently Windows 7/Win2k8 place more trust in "Location Awareness" rather than depending on checking if the IP is in the same subnet or if the computer's in the same domain, etc. It just wants to know that your client thinks it's in the same logical DNS structure that it's on.