The return of UNCHardenedPath problems.

Last week we rolled out some new GPO security settings which made our Windows 10 machines stop being able to process group policy changes.  First we noticed the GPP drive maps had stopped working and when we ran gupdate /force manually it failed citing that it couldn't access gpt.ini for
31B2F340-016D-11D2-045F-00C04FB984F9 (aka the Default Domain Policy).
While researching it we found many articles on how Windows 10 by default has UNC Hardenening enabled and the various patches (MS15-011, MS15-014) had affected many users in GPO environments.  We weren't using user filtering and all of our GPOs had Authenticated users listed with Read and Apply permissions so that wasn't it.  So for testing, we added the registry keys to disable Mutual Authentication on a laptop.

New-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkProvider\HardenedPaths" -Name "\\*\SYSVOL" -Value "RequireMutualAuthentication=0" -Property "String"

New-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkProvider\HardenedPaths" -Name "\\*\NETLOGON" -Value "RequireMutualAuthentication=0" -Property "String"

We were able to run gpupdate /force successfully after that but we didn't like that solution because that meant we'd have to manually update a lot of machines since even login scripts were broken at this point.  That and it just didn't make sense that Microsoft would have implemented all these security controls if they didn't work so we continued researching.  We found the next clue at the end of Sean Greenbaum's post - patch MS16-075 / KB 3161561 which was released in June and purportedly had caused issues for users trying to access SYSVOL shares.

The workaround listed was to set the SmbServerNameHardeningLevel to 0 under
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
on the domain controller servers.  That registry key corresponds to the GPO security policy
Ensure 'Microsoft network server: Server SPN target name validation level' is set to 'Accept if provided by client' or higher
which was one of the settings that we'd changed the week before.  Setting that to Off changes SmbServerNameHardeningLevel to 0.  Once that change was made on the Domain Controller GPO and applied, all of our client issues were resolved.

Ultimately this came down to insufficient testing on our part and it is one of the risks of trying to harden down existing systems.

References;
https://blogs.technet.microsoft.com/askpfeplat/2015/02/22/guidance-on-deployment-of-ms15-011-and-ms15-014/

https://blogs.technet.microsoft.com/askpfeplat/2016/07/05/who-broke-my-user-gpos/

https://social.technet.microsoft.com/Forums/en-US/6a20e3f6-728a-4aa9-831a-6133f446ea08/gpos-do-not-apply-on-windows-10-enterprise-x64?forum=winserverGP

https://community.spiceworks.com/topic/1389891-windows-10-and-sysvol-netlogon

Veeam error after Hyper-V migration

In general I find Veeam backup and replication 9 performs brilliantly once it's configured.  But sometimes infrastructure changes can really throw it for a loop.  I recently had to shuffle several VMs around between Hyper-V hosts using the built-in Move command and afterwards Veeam started throwing errors on some of the VMs. (Task failed:  failed to expand object.  Error:  Cannot find VM on host...)

The main thing they all had in common was they they were configured to use alternate guest OS credentials (which Veeam uses to take the internal snapshots).  In the Veeam GUI these all appear to be tagged by VM Name but what I suspect is that on the back end it's latched on either to the GUID or the host server name so by Moving the VMs it started treating them as new entities.

The fix is a relatively straightforward but manual process of removing them, adding them back from the new associated servers (under Guest Processing, Credentials...), setting the right credentials for them, then hitting OK, then Finish.  That will fix that particular error so you won't see it again on the next run.

Configuring LDAP auth from Palo Alto PA-500 firewalls to Windows 2012 R2 AD servers

For the most part this is covered in the Palo Alto admin guides but if like me you just wind up owning one of these at work and you don't have a bunch of time to decipher it then you might find this useful.  Especially since configuring Palo Altos is a lot like object oriented programming where you have to 'build' out all your components and then chain them together which makes troubleshooting more fun.

LDAP Config (using PanOS release 7.x):

Step 1 -

Device Tab -> Server Profiles -> LDAP.  From here Add a new Server profile, give it a meaningful name like domain-ldap and populate the server list.
Enter in your base DN
Enter in your Bind DN - which in my case I created a dedicated service account and entered it in UPN format as 'accountname@domainname.com'.  Then enter in the password for the account so it'll be able to access the directory.

For AD LDAP, go ahead an uncheck the Require SSL/TLS checkbox.

And Commit your changes

Step 2

Now go to the Authentication Profile (also on the Device Tab) and click Add.
Give it a meaningful name like ldap-authprofile.
Then choose the Server Profile that we created in step 1 from the drop down list.
The Login Attribute should be sAMAccountame.  (no, I don't know if that's case sensitive).
Important - Fill in the User Domain with the NETBIOS name of your domain.  Yes, I know it's 2016 and we're still stuck with it.  It'll make a difference later on if you try to do Group Filtering.
If you're setting up an Allow list then click the Advanced Tab and enter in the LDAP strings for your groups.

And Commit your changes

Optional Step 3 - Group filtering/search

If you're using Group Filtering, make sure to go under User Identification, then to the Group Mappings setup tab and Add those groups in.
Click Add, then choose your Sever Profile that we created in Step 1.
Go to the Group Include List Tab, and drill down to your group.
Note:  if you can't drill down, then you don't have a working LDAP connection.  check your settings and make sure your AD Controllers are listening.  Also, keep in mind that the traffic will be coming From the MGT port on the Palo Alto which may have a different IP.

Click Ok. Commit your changes.

At this point you should have a fully functional LDAP Authentication Profile which you can feed into other objects like Authentication Sequences, GlobalProtect Gateways, etc.

Troubleshooting tips:
The default caching period is about an hour.  If you're doing testing you'll want to force that cache to empty out.  From a console/ssh connection - run
debug user-id refresh group-mapping all
to refresh the LDAP cache.

PanOS 7.x also has a new feature to help you troubleshoot authentication from a command line. Details here:
http://dsg0.com/t/palo-atlo-networks-user-authentication-test-through-cli/273

Good Luck!

Veeam failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing) on hyper-v

We recently abandoned Backup Exec and transitioned to Veeam Backup and Replication and as with most product transitions we ran into a few hiccups.  Aside from having to adjust Shadow Copy space limits on some VMs and Hyper-V hosts, we also ran into snapshot errors.  BE uses agents to handle quiescing while Veeam directly contacts the VM to request snapshot creation.  Some of our VMs were in DMZs and other areas and not on a domain so the default credentials that Veeam was using was not able to authenticate and create snapshots.

The fix for this was to add additional credentials and map them to each of the errant VMs directly.

After we got that sorted out, the rest was a breeze.