How to survive installing Exchange 2010 SP1

In recent years Microsoft, in my opinion, has generally gotten better at putting out service packs that don't cause massive failures. So I was a bit surprised when I saw how many people were having problems with the latest Exchange 2010 service pack. I actually put it off for an extra month after reading some of the forum posts. But this week I revisited it and was able to deploy SP1 on my exchange servers without any hitches.

My usual process for getting ready for an upgrade is to read every forum and blog post about it I can find and to start looking for patterns of what people are running and where they're running into issues. The most useful site I used this time was:
http://msexchangeteam.com/archive/2010/09/01/456094.aspx
After all, no better place to start than a blog run by Microsoft Dev's.

I wasn't very pleased to find that only 1 of the pre-requisite patches I needed was a general release. The others were all what most of us would call beta or pre-release patches but MS swears up and down these have been fully testing and vetted. (Overall still not much comfort when you're installing them on your production servers.)
For my Windows 2008 R2 x64 servers I installed:
979099
982867
983440
981002
979744 - (watch out for an error that pops up for some people who have multiple iis bindings.)

And then I installed the Microsoft Filter Pack 2.0 which was recommended on another forum. Then I rebooted and took a backup of the server.

Then just prior to installing the patch, I stopped these services as recommended in this post:
http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/7c19d906-35ef-4053-bb64-f5a2cab4c008

Exchange Transport Service
System Attendant
Information Store

I also turned off my antivirus program and symantec spam filter.

I then extracted the SP1 patch and then ran setup on my CAS server as a user that had rights to update the domain schema and crossed my fingers. I was pleasantly surprised when it completed without any errors and tested it out to make sure it was working alright.

Then a couple of days later I went ahead and updated the Hub Transport/Mailbox server using the same procedure. So far everything's working fine. Next I'll work on converting my Managed Folder policies to the newer Retention Tag system.

Finally, an electronic switch hook for my plantronics/nortel ip phone.

Since we started rolling out the Nortel IP 200x phones and now the Avaya 11xxe series phones the biggest complaint we've had from our users is that the HL10 lifter from Plantronics keeps falling off. Thankfully, Plantronics has recently released a series of USB EHS devices to integrate existing CS55 and CS70 headsets into our Avaya 1120e IP Phones.

My test configuration:
1. Avaya 1120e
2. APU-7 EHS Cable from Plantronics
3. CS70 headset

Tests:
1. Called into the IP Phone - confirmed that I received a ring warning through the headset
2. Pick up incoming phone call using button on headset - success
3. Turn on headset and get dial tone - success

Sadly, though, they missed one important feature with the new EHS cables - they didn't give us any way to hook up an online indicator light. On the old HL10 it was easy to hook one up but that feature got lost on this new device. Other than that this is a great new solution for us and I won't miss having to deal with all the wiring and points of failure that came with the HL10.

Cisco SLM2008 web managed and STP capable switch review

So I've finally got a solution for my conference room switches. Every once in a while a bonehead user will plug both ends of a cable into the switch and take out a section of the network (STP convergence times heading toward infinity tend to have a detrimental effect on uptime). The larger switches all have Spanning Tree Protocol set up on them so they weren't really affected but I didn't have a cost effective solution for the smaller rooms that needed switches; and preferably the type that don't howl or cost a fortune. Enter in the Cisco Small business smart switch series:
SLM2008 - http://www.cisco.com/en/US/products/ps9996/index.html
They're compact, web-managed, support STP, VLANs, and the 8 port gigabit model only costs $100.

So after having set it up I had to test the bonehead maneuver! As much as the effort went against my better instincts, I took a cable and plugged it into ports 1 and 4. As you can see below, the switch detected the loop and blocked the ports out, thus preventing a cascade of pain to the nearest network closet.



And that's just one less vulnerable flank on my network...

Seagate GoFlex - testing USB 2.0 vs eSATA performance

Recently I picked up some of the new Seagate FreeAgent GoFlex drives with the swappable interface dongles since we've had good luck with the previous generation of drives and we were interested in seeing if we could eek out a little bit more performance by changing interfaces for it. Running virtual machines off external hard drives is slow enough as it is so every 1MB/s you can get is a plus to me. My test platform was a Dell Latitude e6410 and a 500GB GoFlex drive with both the USB 2.0 and eSATA dongles.
First I tried out a few large file copies (500MB to 2GB) and noticed that windows was reporting a higher sustained rate for the USB2.0 but I also noticed a very long delay on the last 1-2% which I'm pretty sure was a cache delay. This delay at the end offset the perceived higher transfer rate that the USB transfer had shown. The eSATA transfers all went smoothly with no lag at the end at all. What can I say, I liked the honesty of the eSATA estimates better.
I then tried out some disk benchmarking tools like CrystalDiskMark (below) for further data and fortunately they all seemed to show the same significant advantage toward the eSATA:



Now I'm by no means an expert but I think eSATA will be the way to go for my Virtual Machines. At the very least the high sequential read/write should make the startup/shutdown for the VMs faster than they currently are under USB. So I shall sent out my little guinea pigs and wait for Murphy's law to kick in...

TMG 2010 and hyper-v guest issue with windows update 80072EE2

I was trying to set up a temporary hyper-v server for demo/training purposes this week and ran into a windows update issue with a 2008 r2 guest on it. I turned on logging at my TMG server just to make sure the packets were getting out okay and then I noticed an odd error that popped up - 0xc0040018 FWX_E_BAD_LENGTH_PACKET_DROPPED - the packet was dropped because its IP length field does not fall within the allowed range or is inconsistent with the actual length.


So I tried looking that one up on the web and came up pretty dry other than the list of error codes from Technet. I decided to change gears and approach it from the other direction and went off in search of the windows update error code 80072EE2 which led me to this blog post:

http://msmvps.com/blogs/thenakedmvp/archive/2009/11/08/windows-update-error-80072ee2-hyper-v-guest-issues.aspx

And then it all clicked together. The offload settings weren't working right on the sub-standard hyper-v server I was using and the effect I was seeing on the firewall was incorrect packet lengths. As soon as I turned off the offload settings on the host server for that network card the windows update problems on the guest OS all went away.

My new favorite spare parts vendor for old Dell Poweredge Servers

(Yes, this post is going to be a vendor plug as you should deduce from the title.)

We all have them lurking in our data centers, those old servers that you just can't seem to kick the users off of and retire gracefully. Inevitably, that server will suffer a hardware failure and if it's a Dell and more than 3 years old the chances are that after a half hour on the phone bouncing around they'll finally just admit that they don't have the part and have no idea when it'll ever be in stock again. Oh, but if it ever did magically appear it'll cost $800.

While digging through the forums I noticed another user had recommended Velocity TechSolutions http://www.velocitytechsolutions.com/ I'm always a little skeptical of new sites but being in a pinch I wasn't going to be picky. After a quick search through the site I found the parts kit that I needed for $99. I went ahead and ordered it with overnight shipping and had a fedex tracking confirmation within a few hours and the parts were here the next morning. Overall I've got a good first impression of them.

The Microsoft IT Site Delete Capture Feature 1.0 that wouldn't die

So merely a day after the marathon weekend of getting my MSSX 2008 (wss 3.0 sp2) instance up to Sharepoint 2010 Server, an old add-on came back to haunt me. The Microsoft IT Site Delete Capture Feature 1.0 (MS.IT.SiteDeleteCapture) whose installation was long ago lost in a fiery server crash and whose features had long been disabled. Additionally, the install files for it have long disappeared from the face of the earth as all the links to it are dead. http://technet.microsoft.com/en-us/library/cc706867%28office.12%29.aspx (Thanks codeplex for ditching the old junk! LOL). What really makes it great is that it never came up as an issue in the preupgradecheck report.

Anyway I had a user report that they couldn't delete a subsite. Lo and behold the user was right. I used the trusty ULS viewer to decode the wonderfully cryptic Microsoft Correlation ID and discovered the culprit.



It is experiences like these that have converted me over to the philosophy of NEVER installing any third party addons to a sharepoint/wss/mssx install that didn't come out of the DVD or from the Microsoft update site. Or at the very least, plan on leaving there long before the next major upgrade.

So I ran my old favorite utility: http://featureadmin.codeplex.com/ (awesome utility btw). But it didn't show up there. I tried a few more tools and came to the conclusion that as far as sharepoint was concerned, it wasn't installed which meant that I had some phantom Event Hook of some type lingering around. At this point my two remaining choices were A) Open up a Microsoft PSS ticket or B) Roll the dice and try something crafty and stupid.

My plan B was to just plop on the newer version of the IT site capture following the age old wisdom of the best way to cure a hangover (more alcohol btw if you missed the reference). I found it at the redirected link for the old product under the new auspicious title of Sharepoint Governance. My theory was that it was the successor to the old product, and it would probably Event hook in a similar way, that it was worth a shot. Worst case scenario it would just be one more thing for them to fix for me if I wound up with PSS.

Cue the drum roll....



And we're back in business. And yes, as a sanity check I did check and see to make sure the site was gone and I also created a new site and deleted it as well. So far so good!

2008 multi-homed dns server failing simple query but otherwise works fine

So my dns server was working fine, resolved queries, updated records, etc. But for some reason it kept failing the built in "Simple Query" and "Recursive Query" tests. After much messing around I discovered that it was trying to query the first interface it found. In this case it was the interface that I had manually excluded from the list because I didn't want DNS listening on it. I wound up changing the binding order for the network interfaces to fix the problem. After swapping them around I restarted DNS and voila the built in Monitoring started showing 'Pass" instead of "Fail".

List of interfaces, notice the non-listening one shows up first.

Under Network and Sharing, Advanced Properties, change the bind order.
Restart DNS and try again.