Windows 2008 R2 backup exec and failure occurred accessing the Writer metadata - Workaround

Updated 9/7/09

Nothing tramples the joy of playing with a new operating system faster than finding out that your vendor is being a deadbeat and hasn't put out a compatible release yet. You'd think that out of the army of programmers that Symantec has that they'd have at least one technet or msdn subscription and that they'd have started working out compatibility issues in the meager half year that the betas were available. I was also amused to find that on their forums some of their staff didn't realize that the RTM was out yet for Windows 7 and 2008 R2... But I digress.

So you're using Backup Exec 12.5 and trying to backup a Windows 2008 R2 RTM server using the Advanced Open File option and you get this error:

V-79-57344-65225 - AOFO: Initialization failure on: "\\MyServerName\System?State". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).
Snapshot provider error (0xE000FEC9): A failure occurred accessing the Writer metadata

• Option 1: Wait a month or so till a hotfix comes out.
• Option 2: Wait until Backup Exec 2010 comes out with official support for R2.
• Option 3: Fix the VSS issue that's causing it in the first place!

During the installation of Windows 2008 R2 RTM, it creates a Recovery Partition that's about 100MB. When the AOFO agent kicks in, it works with the VSS providers in the operating system to create snapshots. However, VSS really doesn't like those tiny partitions like the 100MB System Reserved (Recovery) partition. So at this point you have two choices.

• A) Wipe the partition out. (Note, if you used Diskpart to setup the drive instead of the windows 2008 setup program, this won't exist anyway.)
  
• B) Find a workaround for the VSS snapshot.

I didn't really want to do option A yet as I'm not fully sure if that'll have any impact down the line so I decided on option B.

UPDATE: Some of you reported success with just assigning the partition a drive letter. Try it and if it works for you, then don't bother with the vssadmin parts.

I got pretty familiar with the VSSADMIN command while working with Hyper-V and backups so I knew that it could be used to redirect VSS snapshots to larger partitions. The problem I ran into is that it didn't like the fact that the System Reserved partition didn't have a drive letter. So I did the quick fix and used Disk Management to assign it a random drive letter - in this case P:



Then a quick drop to a command prompt and run vssadmin list volumes

C:\Users\Administrator>vssadmin list volumes
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Volume path: P:\
Volume name: \\?\Volume{a2b716d3-8c1f-11de-a5ed-826d6f6e6973}\
Volume path: C:\
Volume name: \\?\Volume{a2b716d4-8c1f-11de-a5ed-826d6f6e6973}\
Volume path: D:\
Volume name: \\?\Volume{75c2418c-8c0e-11de-ae3c-001143dd2544}\

You'll note there's an entry for all your partitions. Now we set up a ShadowStorage for P:\ (100MB partition). ShadowStorage basically sets aside space on a volume to store snapshots of a volume. In this case I'm going to store snapshots of P: on D:

vssadmin add shadowstorage /For=P: /On=D: /MaxSize=1GB

And you have to put a MaxSize so I picked 1GB.

Now run vssadmin list shadowstorage to confirm the link has been set up.

C:\Users\Administrator>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Shadow Copy Storage association
For volume: (P:)\\?\Volume{a2b716d3-8c1f-11de-a5ed-826d6f6e6973}\
Shadow Copy Storage volume: (D:)\\?\Volume{75b2419c-8c5e-11de-af3b-001143dd23
44}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: 1 GB (4%)

If you have any other volumes configured for Shadow Copies you'll also see them listed there. (i.e. If you enabled "Previous Versions" for a file share, etc)

At this point you're done. I was able to do a successful backup of the server with the AOFO (Advanced open file option) enabled after making this change. My backup seemed a bit slow but it is an older server so I can't be sure if speed was a machine issue or an R2/Symantec issue.

Windows 7 RTM, SQL 2008 dev edition x64 and invoke issues

While installing Windows 7 x64 is a breeze, putting SQL 2008 developer edition on top wasn't. Upon my first attempt the application compability warning popped up saying to install SQL 2k8 SP1 afterwards. Which would be fine if the install didn't die right after that. Or if MSDN had a already slipstreamed SP1 version on the download site...

So, for round 2 I used "Procedure 1" of this KB:
http://support.microsoft.com/kb/955392
Which basically walked me through download/extract the SP1 file and trying to launch setup with the PCUSource flag.

Setup.exe /PCUSource=C:\SP1

This did allow me to progress further and then I wound up with this error:


Invoke or BeginInvoke cannot be called on a control until the window handle has been created

So I dug around ye olde web a bit more and tried installing the SQLSupport.msi from the extracted SP1 files. That didn't work or at least not by itself. Another forum suggested rebooting but that didn't do squat either.

Finally, I had to resort to using "Procedure 2: Creating a merged drop" from the KB listed above. This time we had success.



I was going to reapply SP1 after the install finished as a just in case, but the SP1 patcher told me the machine was already updated and wouldn't let me proceed. So we'll call it a day.

How not to get stuck at Precopy preparation during a Dell system build

So I unpacked the mini cardboard crate that my new Dell Poweredge R700 came in and did the usual inventory (BTW, it's a sweet, sweet machine). I noticed that they hadn't shipped the usual Dell Openmanage CD pack and since I was going to do some testing, I kinda needed it to do some OS reloads. I went to the website and downloaded the two ISO files (1.9GB and 1.8GB) thinking they were two different DVDs. The nifty thing about ISO files is that you can split them up any way you want and your dvd burning software will burn DVDs for you no matter how broken the result may be.
Anyway, I popped in the first DVD and booted off of it and then choose the System Builds and Update Utility. Then went through and choose my 2008 x64, time zone, etc and told it to apply. Then it stopped dead 15% into it at Precopy preparation.



Then began the troubleshooting. Suffice it to say that it wasn't any of the usual things. So I went back to the website to look for an older version of the OpenManage DVD. While digging through, I noticed an interesting comment buried down under 'Additional Information'.

To address a browser limitation around downloading large files (see Microsoft KB article 298618: You cannot download files that are 2 GB or larger - http://support.microsoft.com/kb/298618), the Dell Systems Management Tools and Documentation DVD as a single ISO file is no longer available for web download. You can do one of the following to get the content:

1) If you recently bought a server, please use the DVD that shipped with your hardware.

2) Download the two ISO file segments to a new, empty folder and concatenate them. Create a single DVD image file using the following commands:
Windows: copy /b OM* OM_610_SMTD_A00.iso
Linux: cat OM* > OM_610_SMTD_A00.iso

CONCATENTATE. Yes, this critical piece of information was neither located under "Description" nor was it under "Important Information". Nay, it was located under "Additional Information" due to it not being important... Still at the end of the day it technically qualifies as a RTFM moment.

Imagine my surprise when I actually followed the instructions and then re-burned the DVD, my install worked correctly.

The unequivocal joy of sharepoint and one way trusted forests

Sharepoint is one of those products that's great once it's installed and configured. The configuration of Sharepoint, however, remains a real pain...

Today's challenge was setting up a WSS 3.0 server in the testing lab. The testing lab has a separate AD forest that only has a one-way trust to the production forest. The requirement was to have the WSS 3.0 server be part of the LABTEST domain AND be able to add users from both PROD and LABTEST to the application. Now that seems simple enough since the server already sees both domains as evidenced by the logon drop down box showing both domains. However, as I found out that doesn't mean that the web app will see both as well...

Finding the right command to run was a relatively easy google search which sent me to technet. Getting the syntax right and figuring out how to use the command correctly, now that was the fun part. With the assistance of these two blogs I got it to work:

http://blogs.msdn.com/joelo/archive/2007/03/08/cross-forest-multi-forest-configuration-additional-info.aspx

http://blogs.msdn.com/sharepoint/archive/2006/03/15/552331.aspx

First, in several discussion groups I got differing answers over whether or not the Sharepoint Application Pool Identity needed to be set to "Network Service" or as a domain user account in the domain (in my case, the LAB domain). I used a domain user account myself but had to make changes to the DCOM because my pool wouldn't start. (Component Services - Computers -> My Computer -> COM+ Applications -> DCOM CONFIG -> IIS WAMREG -> Properties -> Security Tab -> Edit Launch and Activation and just give the domain user permissions).

Next it's time to go to a command prompt and go to C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN . Now don't do what I did and think that just because you don't have a full web farm that you can skip the first instruction

stsadm.exe -o setapppassword -password SomeRandomPassword

Literally, it doesn't matter what is set to, as long as it's the same on all your front end servers. Even one lonely standalone installation...

The second command is a bit on the long side. The full syntax I ran (names and passwords have been changed for security purposes).

stsadm.exe -o setproperty -url http://WSSVM1 -pn "peoplepicker-searchadforests" -pv "forest:AD.PROD.COM,TrustUser,3t9sz9$b20pz;forest:LAB.LOCAL;domain:AD.PROD.COM,TrustUser,3t9sz9$b20pz;domain:LAB.LOCAL"

Where AD.PROD.COM is the FQDN of my forest and root domain and LAB.LOCAL is the FQDN of the forest and root domain. (keep in mind LAB.LOCAL is the domain that the WSS server is joined to. You'll notice that I had to use a domain user account in the Trusted domain in order to be able to search it since it's only a 1 way trust. Also, while digging I found on one of the forums that you DO need to include the domain the server is joined to as well as the domain you want to add to the search. I'm not really sure if it's necessary to include both the forest: and domain: for each but it works this way so I'm sticking to this method.

On a related note, there's also an alternative way to set this up that involves setting up shadowed, non-login accounts in the resource domain that map to the real users in the production domain. It's a bit more than I needed for this project but you might find it useful.

http://blogs.msdn.com/sharepoint/archive/2006/03/15/552331.aspx

vmrun - the grown up pain in the b** replacement for vmware-cmd

Back in the VMWare Server 1.x days stopping and starting up VMs from the command line was easy with 'vmware-cmd'. Now with Version 2.0 on my win2k3 boxes, I was forced to learn how to use the replacement - VMRUN.

I do understand that those who use the 'real' VMWARE products - ESX, vSphere, etc are quite used to using VMRUN. But if you're used to the old easy way and you're still stuck on the free versions, it's a bit tricky to get to work.

1. First off, you'll want to make sure your %PATH% variable is updated for the path to the vmrun command. ("C:\Program Files (x86)\VMware\VMware Server" on my x64 box)

2. Now open a command prompt and enter in vmrun list. This will show you a listing similar to this:

Total running VMs: 5
[standard] MYWEB2B/Win2k3R2STD_vmsrv2.0.vmx


where [standard] is the name of the default datastore in VMWare 2.x, MYWEB2B is the immediate folder under that in the file system, and then you have the filename. IT IS CASE SENSITIVE. The whole @#$% thing. If you mess that up, it will just tell you that "The virtual machine cannot be found".

3. Now that we know how to reference the vm files properly, now we need to specify the -T, -h, -u, and -p parameters. For some unknown reason, they don't appear to have a 'just run it as the server I'm sitting on and as the users I'm running this as' setting'.
Since you're using VMWare Server 2.0, you'll use -T server -h https://YourServerNameOrIP:8333/sdk (yes, it needs the sdk at the end)
and then provide a username/password (-u,-p) that is part of the administrators group on the host machine. (or if you've setup custom permissions in vmware, use one of those).

At this point, we should be able to construct this command to stop a VM gracefully:

vmrun -T server -h https://MyVMHostServer:8333/sdk -u vmadmin -p thepassword stop "[standard] MYWEB2B/Win2k3R2STD_vmsrv2.0.vmx" soft

You'll note that I introduced the 'stop' and 'soft' parameters. I'll give you three guesses what 'stop' does. The shutdown type option 'soft' will run the shutdown scripts for you for the VM to gracefully power it down. If you wanted to drop it uncleanly, just use 'hard' instead.

4. To Start it back up:
vmrun -T server -h https://MyVMHostServer:8333/sdk -u vmadmin -p thepassword start "[standard] MYWEB2B/Win2k3R2STD_vmsrv2.0.vmx"

Now combine this will some old style batch files and robocopy or xcopy and you've got a cheap way to make VM backups using Task Scheduler.

Note 1: Sometimes it just doesn't like netbios names.

Additional References:
VMRUN reference:
http://www.vmware.com/pdf/vix162_vmrun_command.pdf

Tips and Tricks for vmware server 2.0:
http://www.google.com/url?sa=t&source=web&ct=res&cd=1&url=http%3A%2F%2Fcommunities.vmware.com%2Fservlet%2FJiveServlet%2FpreviewBody%2F9394-102-2-6307%2FVMware%2520Server%25202.0%2520Tips%2520and%2520Tricks.pdf&ei=O4ItStbKI5WEtweRtM2vCA&rct=j&q=VMware+Server+2.0+Tips+and+Tricks.pdf&usg=AFQjCNH0ZvlIVPIrc8N_UQ64Y1efSoIO_Q

SoftOSD and Dell latitude laptops with the Nvidia chipset

I haven't confirmed it yet (and you'll understand why after I explain it) but I was told that the Dell Client Manager agent that comes with the Dell Management Console was causing issues on some D620 units that had the Nvidia chipset. On those units the entire LCD stopped working and require Dell to replace the motherboard and LCD. While this series of laptop has been plagued by Nvidia related issues, I was assured that these units had been running fine for a few years already and the sheer quantity of them failing at the same time was unlikely. The current suspect is the SoftOSD component which gets installed as part of the Dell Client Manager Agent. I looked up this component and it appears to be directly related to video card/output manipulation which only adds to our suspicions.

Also, these units seemed to all have been docked for at least some of the time prior to their failure and caused odd resets on the external monitor.

While this hasn't been verified yet, I would recommend that you uninstall the SoftOSD component off any D620 with the Nvidia chipset until we know one way or another. I'm not exactly willing to offer up any of my D620's for testing sacrifice. :)

KB 968078 (MS09-016) broke RDP on my ISA 2006 server

I really shouldn't be surprised since the last time I installed a hotfix for my ISA 2006 SP1 (win2k3 sp2 with all offloading, etc turned off) it broke RDP as well. I've already seen other forum hits on this now as well. I uninstalled it for now as being without RDP access to that server is not an option right now. Others have reported VPN problems and problems starting up standard edition with 4+ cpu cores. I'm not going to fool with it until I see a new patch come out or a resolution for this problem.

some other references to this issue:

http://forums.isaserver.org/m_2002085188/mpage_1/key_968078/tm.htm#2002085188

http://blogs.technet.com/isablog/archive/2009/04/18/ms09-012-and-isa-server-standard-edition-14109-failures.aspx

Symantec Mail Security for Exchange Named Piped Error updating license

Yes, you read that right, "The Named Piped could not be found" and because of that misspelling, you won't find the answer on their knowledge base.

I was trying to Add my new updated license files through the management console for Symantec Mail Security for Exchange and I got that error. The real error should read named pipe instead of piped and that's how you'll find it in their KB. When you get this cool error, you'll also notice the checkbox for "Enable Premium Antispam" gets whacked and your users start complaining about spam.

http://service1.symantec.com/support/ent-gate.nsf/docid/2007072606354254

Basically they'll have you stop both the services and manually wipe out the existing license files that are there. Then restart the services and try to add them again. Fortunately, this actually worked for me. Make sure to re-enable the premium antispam checkbox if you have that service.