Installing 2008 Server Core - VMWARE Server 1.0x

When you go to create the new virtual machine, choose the "Vista" option for OS type (It's about as close as you're going to get). Insert the 2008 CD or mount the ISO as a CD instead. Server Core installs without a hitch. Click Other when the GUI comes up, login as Administrator and it'll force you to choose a new password.

To get VMWARE Tools installed, do the usual step in the VMWARE console of clicking VM-> Install VMWARE Tools which will mount d:\ to the vmware tools iso. Go to the command prompt in the VM and cd to d: and then run VMWare Tools.msi (add a /qn for silent install).

To rename the server, first run HOSTNAME to confirm the current name, then run
netdom renamecomputer InsertOldNameHere /NewName InsertNewNameHere
*Note, this will require a reboot to kick in fully. (Shutdown /r for those who still haven't gotten their old GUI-less legs back). Shutdown /r /t 0 for the impatient.

To update the license key for the server:
slmgr -ipk InsertKeyHereWithDashes

Then slmgr -ato to activate windows (may take several minutes)

To enable Remote Desktop (hehe, seems odd for a gui-less install but yet it does have its uses). This command should add firewall rules for RDP inbound for you.

"Cscript %windir%\system32\SCRegEdit.wsf /ar 0" Enable for vista/2k8 clients
"Cscript %windir%\system32\SCRegEdit.wsf /cs 0" Enable for older clients
This worked fine for me until I added it to a domain at which point RDP stopped accepting connections. So I added a manual rule for it:
netsh firewall set portopening tcp 3389 "Remote Desktop"

Now if you're like me and the 640 * 480 resolution is killing you and you're feeling brave there is a way to change it. Run regedit from the command prompt. . Navigate to the following keys:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Video\GUID\0000\DefaultSettings.XResolution
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Video\GUID\0000\DefaultSettings.YResolution
Just to make this confusing, there will be multiple GUIDs. Pretty much check them all and see which one has the XResolution set to Hex 280 and YRes set to 1e0. Figure out the Hex equivalent of what your desired resolution is (use calculator on another box if needed). I just went with 320*258 Hex (800*600 decimal) but you could probably run 400*300 (1024*768). Close out the registry editor and reboot for changes to kick in.

At this point you've got a good base VM image to backup and then start playing around with.

Tips from the guys at microsoft:
http://blogs.technet.com/server_core/