The great wonderware bug hunt

I have ranted previously about a few bugs in System Platform 2017, but one in particular was quite interesting so I thought I would take the time to write it up as I learnt a few things along the way.

The Symptoms

After installing System Platform, other users that were not administrators, and not the use that installed System Platform could not log in.  Or rather they could log in, but the shell (explorer) would not load.  Some things that I tried:

  • Checked that all the shell entries were actually set to explorer.exe.  They were.
  • Checked that explorer was launching with procmon, which is was, but it quit early or got stuck without doing anything useful.  Couldn't see why.
  • Installing SP2017 on a clean, non domain joined machine.  Couldn't reproduce.
  • Insalling SP2017 on a clean, domain joined machine. Couldn't reproduce.

At this point I'm thinking one of our group policies was causing some kind of issue, but nothing was jumping out at me.  Around this time the GCS rep that I was in touch with about this bug got in touch and suggested it might be a permission issue, so I traced explorer.exe again, and filtered by ACCESS DENIED.  This time I got two or three entries that stood out.  So I took a look at them and compared them to another machine, and the obvious difference was that .\Users was missing from my SP2017 machine (and that aaUsers, aaPowerUsers, aaAdministrators also had permissions).  So it was now looking quite convincing that the System Platform installer was messing with the permissions, but why was it doing it on the machines in my domain, but not on other machines?

At this point I traced the entire installation with procmon, which generated some 40 million events and several GB of data.  After searching the data for permissions changes on the affected keys, I narrowed in on the program aahsecurity.exe.  I also enabled object auditing so that I could see exactly what permissions aahsecurity was changing and it clearly showed that the permissions on HKCR\CLSID\{xxx...} were having their inherited Users permissions removed, the question was... why?

HKCR

Something started to smell funny when I had a look at the permissions for HKCR\CLSID and noted that the .\Users didn't have permissions on this key, even though it should have been inheriting that permission from its parent HKCR.  Many will know that HKCR is actually a merging of HKLM\Software\Classes and HKCU\Software\Classes, so I began to wonder how the permissions are actually assigned when they could potentially be different from the different stores?  When comparing HKLM\Software\Classes\CLSID and HKCU\Software\Classes\CLSID  I noted that the HKCU not only existed where it didn't on the other machines I tested, but it didn't have the Users permission.  So it would appear that HKCR\CLSID has the permissions of the some franken combination of the HKLM and HKCU keys, but the subkeys still inherit the permissions from there physical parent.  This all seems quite logical.

So, aahsecurity was taking the permissions of HKCR\CLSID, adding aaUsers, aaPowerUsers and aaAdministrators and forcing inheritance down on all children which obliterates the .\Users permission previously inherited from HKLM\Software\Classes\CLSID.  What is should be doing, is applying these permissions to HKLM\Software\Classes\CLSID instead.  All of the above also applies to HKLM\Software\Classes\Wow6432Node\CLSID as well.

The fix is to delete the HKCU\Software\Classes\{Wow6432Node\}CLSID either before installation or if done after installation you will need to run aahsecurity again. It might be easiest to create a new user to do this because if your HKCU\Software\Classes\{Wow6432Node\}CLSID is not empty it will cause other problems.

However, one question remains:  aaUsers contains BUILTIN\Users.  So why don't the permissions propagate through the groups?  This is something I don't know and I have an open question on server fault if anyone knows why https://serverfault.com/questions/874319/registry-permissions-with-nested-groups

UPDATE: Wonderware has released a technote regarding this issue, and hotfix L00146083 is available.

 

| September 25th, 2017 | Posted in SCADA |

Leave a Reply