Wednesday, August 17, 2016

The return of UNCHardenedPath problems.

Last week we rolled out some new GPO security settings which made our Windows 10 machines stop being able to process group policy changes.  First we noticed the GPP drive maps had stopped working and when we ran gupdate /force manually it failed citing that it couldn't access gpt.ini for
31B2F340-016D-11D2-045F-00C04FB984F9 (aka the Default Domain Policy).
While researching it we found many articles on how Windows 10 by default has UNC Hardenening enabled and the various patches (MS15-011, MS15-014) had affected many users in GPO environments.  We weren't using user filtering and all of our GPOs had Authenticated users listed with Read and Apply permissions so that wasn't it.  So for testing, we added the registry keys to disable Mutual Authentication on a laptop.

New-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkProvider\HardenedPaths" -Name "\\*\SYSVOL" -Value "RequireMutualAuthentication=0" -Property "String"

New-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkProvider\HardenedPaths" -Name "\\*\NETLOGON" -Value "RequireMutualAuthentication=0" -Property "String"

We were able to run gpupdate /force successfully after that but we didn't like that solution because that meant we'd have to manually update a lot of machines since even login scripts were broken at this point.  That and it just didn't make sense that Microsoft would have implemented all these security controls if they didn't work so we continued researching.  We found the next clue at the end of Sean Greenbaum's post - patch MS16-075 / KB 3161561 which was released in June and purportedly had caused issues for users trying to access SYSVOL shares.

The workaround listed was to set the SmbServerNameHardeningLevel to 0 under
on the domain controller servers.  That registry key corresponds to the GPO security policy
Ensure 'Microsoft network server: Server SPN target name validation level' is set to 'Accept if provided by client' or higher
which was one of the settings that we'd changed the week before.  Setting that to Off changes SmbServerNameHardeningLevel to 0.  Once that change was made on the Domain Controller GPO and applied, all of our client issues were resolved.

Ultimately this came down to insufficient testing on our part and it is one of the risks of trying to harden down existing systems.


Friday, June 17, 2016

Veeam error after Hyper-V migration

In general I find Veeam backup and replication 9 performs brilliantly once it's configured.  But sometimes infrastructure changes can really throw it for a loop.  I recently had to shuffle several VMs around between Hyper-V hosts using the built-in Move command and afterwards Veeam started throwing errors on some of the VMs. (Task failed:  failed to expand object.  Error:  Cannot find VM on host...)

The main thing they all had in common was they they were configured to use alternate guest OS credentials (which Veeam uses to take the internal snapshots).  In the Veeam GUI these all appear to be tagged by VM Name but what I suspect is that on the back end it's latched on either to the GUID or the host server name so by Moving the VMs it started treating them as new entities.

The fix is a relatively straightforward but manual process of removing them, adding them back from the new associated servers (under Guest Processing, Credentials...), setting the right credentials for them, then hitting OK, then Finish.  That will fix that particular error so you won't see it again on the next run.

Tuesday, February 9, 2016

Configuring LDAP auth from Palo Alto PA-500 firewalls to Windows 2012 R2 AD servers

For the most part this is covered in the Palo Alto admin guides but if like me you just wind up owning one of these at work and you don't have a bunch of time to decipher it then you might find this useful.  Especially since configuring Palo Altos is a lot like object oriented programming where you have to 'build' out all your components and then chain them together which makes troubleshooting more fun.

LDAP Config (using PanOS release 7.x):

Step 1 -

Device Tab -> Server Profiles -> LDAP.  From here Add a new Server profile, give it a meaningful name like domain-ldap and populate the server list.
Enter in your base DN
Enter in your Bind DN - which in my case I created a dedicated service account and entered it in UPN format as ''.  Then enter in the password for the account so it'll be able to access the directory.

For AD LDAP, go ahead an uncheck the Require SSL/TLS checkbox.

And Commit your changes

Step 2

Now go to the Authentication Profile (also on the Device Tab) and click Add.
Give it a meaningful name like ldap-authprofile.
Then choose the Server Profile that we created in step 1 from the drop down list.
The Login Attribute should be sAMAccountame.  (no, I don't know if that's case sensitive).
Important - Fill in the User Domain with the NETBIOS name of your domain.  Yes, I know it's 2016 and we're still stuck with it.  It'll make a difference later on if you try to do Group Filtering.
If you're setting up an Allow list then click the Advanced Tab and enter in the LDAP strings for your groups.

And Commit your changes

Optional Step 3 - Group filtering/search

If you're using Group Filtering, make sure to go under User Identification, then to the Group Mappings setup tab and Add those groups in.
Click Add, then choose your Sever Profile that we created in Step 1.
Go to the Group Include List Tab, and drill down to your group.
Note:  if you can't drill down, then you don't have a working LDAP connection.  check your settings and make sure your AD Controllers are listening.  Also, keep in mind that the traffic will be coming From the MGT port on the Palo Alto which may have a different IP.

Click Ok. Commit your changes.

At this point you should have a fully functional LDAP Authentication Profile which you can feed into other objects like Authentication Sequences, GlobalProtect Gateways, etc.

Troubleshooting tips:
The default caching period is about an hour.  If you're doing testing you'll want to force that cache to empty out.  From a console/ssh connection - run
debug user-id refresh group-mapping all
to refresh the LDAP cache.

PanOS 7.x also has a new feature to help you troubleshoot authentication from a command line. Details here:

Good Luck!

Saturday, February 6, 2016

Veeam failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing) on hyper-v

We recently abandoned Backup Exec and transitioned to Veeam Backup and Replication and as with most product transitions we ran into a few hiccups.  Aside from having to adjust Shadow Copy space limits on some VMs and Hyper-V hosts, we also ran into snapshot errors.  BE uses agents to handle quiescing while Veeam directly contacts the VM to request snapshot creation.  Some of our VMs were in DMZs and other areas and not on a domain so the default credentials that Veeam was using was not able to authenticate and create snapshots.

The fix for this was to add additional credentials and map them to each of the errant VMs directly.

After we got that sorted out, the rest was a breeze.

Thursday, February 19, 2015

Bitlocker could not be enabled - Dell Latitude 7440

Sometimes it just doesn't pay to disable stuff in BIOS.  I recently had problems enabling Bitlocker on some Latitude 7440 units.  After the initial reboot the error would pop up saying that the bitlocker encryption key cannot be obtained.  So I checked the usual suspects - clearing the TPM, making sure the TPM was recognized in device manager, etc.  And then I remembered the USB settings that we'd changed to lock down the laptops more.

Specifically under System Configuration, USB Configuration, "Enable Boot Support" which had been disabled just to make sure our users wouldn't be able to boot off USB devices.  I wouldn't have equated that bitlocker error with that setting but as soon as we undid it on the laptops we were able to enable bitlocker.

Monday, November 3, 2014

Yet another fix for Lync 2013 or Skype 2016 - the server is temporarily unavailable error

I ran into this one on a 365 hosted, AD FS authenticated system with users running Lync 2013. Several users just suddenly started getting this error at random but it would only affect that profile on each machine.  If the same user logged in from a local profile or another domain profile then Lync would log in fine.  And if you torched the whole cached profile and forced a new profile load, Lync would work fine as well.
After having tried multiple Microsoft provided resolutions of deleting Lync profiles, temporary folders, cryptography keys, accountprofile.dat, delete sign-in info, etc., their support team had us kill the Lync process completely and then delete the whole
HKCU\Software\Microsoft\MSOIdentityCRL key
and then start up Lync.  And then it started working properly and we replicated the fix on the remaining affected profiles.

Friday, May 23, 2014

Lync 2013 android client error connecting to ADFS 3.0 federated 365 service

So after our migration to ADFS 3.0 from the old ADFS 2.0 servers my Android based Lync users started getting we can't sign you in, please try again errors during login.  After digging around I found this forum entry from Jeffr.M which points out that the Lync android app has an issue with servers that can support multiple certificates on the same IP.

The following command adds a new default catch-all listener to your server.  If you're using a Web Application Proxy like I am you'll want to run this on that server as well.

netsh http show sslcert

The command above will show you all the listeners and their associated certificate hashes and application IDs.  You'll need those for the next step.

netsh http add sslcert ipport= certhash=INSERTHASHHERE appid='{INSERTAPPIDHERE}'

Note the ticks around the appid.  Powershell sometimes eats curly brackets so you'll get an error if you don't use the "'" marks.  More info here

Note 2: If you're thinking it's easier to just copy/paste the certificate hash from the MMC Certificates panel - Don't. That method often introduces hidden characters which will take forever to debug.

After you do that on your ADFS 3.0 and WEP servers, restart the ADFS services on them and then your Android Lync clients will start working again.

On a related note, if your Onedrive authentication isn't working - try disabling the /adfs/services/trust/2005/windowstransport endpoint.  (disable on proxy if using a proxy or just disable both modes just in case).  There's a bug with the windowstransport endpoint in ADFS 3.0 and Onedrive authentication.

Tuesday, January 7, 2014

Fix for Wake after sleep freeze on Dell Latitudes

This turned out to be an issue with the O2Micro SD/MMC drivers on the E6420/E6430 units that we had. After adding new drivers to our MDT server I started getting reports from users stating that their laptops were completely freezing up after waking from sleep - no response to keyboard, mouse, etc.  No mini dumps were generated, powercfg - energy didn't show any major issues, and event viewer was useless.  It was occuring both on Windows 7 and Windows 8.1 builds.

It wasn't until after I started disabling hardware components in Device Manager that I found a correlation between disabling the O2Micro SD/MMC controllers and it being able to wake from sleep.  (I rebooted after each diagnostic test just to make sure all changes were in full effect)

Installing an older version of the driver and rebooting fixed the problem on all the laptops that were having the hang issue.  Of course, just disabling the SD/MMC controllers is a fine fix too.