11.22.17

System Platform 2017 Update 1

Here are some things that are still not fixed in Update 1:

  • Graphic overrides do not work in InTouch or OMI, unless they are pre-placed into a parent graphic.  This is my biggest pet peeve, I have 100s of symbols that are simply a container for an object graphic.  I honestly can't imagine what possible reason there is for deciding that this functionality is not important.
  • Graphics that are affected by wizard options do not propagate correctly in InTouch.  This functionality has not been implemented yet, and is of low importance apparently.
  • Exporting object to csv often results in errors on reimport.  Might have to put in a SR for this.

Here are some new things:

  • The SDK!  It's actually quite easy to use, although the samples that are distributed with the SDK might seem complicated if you aren't familiar with MVVM and WPF.  I have already ported the aaAttributeBrowser app to work with this.
  • Apps work much better.  Breadcrumb, Titlebar and Tree navigation apps in particular are quite a bit better than they were previously.
  • Lots of bugs fixed in the map app, but a still few present as well.

Some warnings about migrating an existing galaxy: when I migrated from 2017 to 2017 update 1, my objects got pretty messed up.  All my object instances gained all the attributes of their parent templates, even though most were disabled via wizard options.  When I exported all of my instances via galaxy dump and reimported, lots of them got imported with the wrong parent template which wasn't at all useful.  I ended up hosing all my objects and templates and importing them from my previous backup and things were well from there.

| Posted in SCADA | No Comments »
11.16.17

protplot update

Protplot has been updated with a couple of new features:

  • Improved UI - pretty icons, fixed several annoying bugs
  • Printing - can print to A3/A4
  • Added a pickup multiplier for inverse time curves
  • Added some more fuses curves
  • Now uses .Net Standard (for future windows store aspirations)
ProtPlot

ProtPlot

Source and binaries on github.

| Posted in Software | No Comments »
10.20.17

Running RTV and RTU Station on Windows 10

The SY-1101211D release of RTV and RTU Station does not support Windows 10,  which is no surprise since it took several years for Schneider to add support windows 7.   There are a couple of reasons that the standard installation doesn't work -

  • The included version of Java doesn't support Windows 10
  • Internal windows compatibility settings cause the RTU Station precheck process to elevate, which causes bash to crash for reasons I haven't bothered to investigate

To fix the first problem:

  1. Download the latest version of Java (8u151 at the time of writing) and install it - make sure you download the 32 bit version even if you are on a 64 bit system!
  2. Change the launcher.ini file in C:\Program Files (x86)\Foxboro\Remote Terminal Viewer, and edit the JRE line to JRE=C:\Program Files (x86)\Java\jre1.8.0_151\bin\javaw
  3. Open up the permissions for the C:\Program Files (x86)\Foxboro\Remote Terminal Viewer folder.  On my system the installer messed all the folder permissions up - we need the Users group to have Modify and Write permissions, so add those in if they don't exist.  If the splash screen never goes away after you open RTV, then these permissions are not correct.
  4. Done!  This is enough if you just have RTV installed.

If you have RTU station installed as well:

  1. In C:\Program Files (x86)\Foxboro\RTU Station\launcher.ini, change the JRE as above.
  2. Check the permissions for C:\Program Files (x86)\Foxboro\RTU Station\ as above.
  3. If after doing step 1 RTU Station always fails the precheck, try changing the PROGRAM line to
    PROGRAM=cmd /c "set __COMPAT_LAYER= & ..\cygwin\bin\bash.exe -c "cd scripts && ./precheck.sh > ../startup.log 2>&1" "
    

    This wraps up bash in a cmd instance, where we disable the compatibility environment variable that causes it to auto-elevate.

  4. If RTU Station still fails quietly (getting stuck on the splash screen), then change the line ARG3=-Xmx1024m to ARG3=-Xmx512m (credit to the chap from the 2017 user group conference for coming up with this one!)

A special note - the backup\restore utility seems to not support the compressed backup format so if your old data backups don't restore then try to unzip them manually first.

| Posted in Foxboro | No Comments »
10.20.17

compiling SALL calcs for SCD6000 without WindRiver

Schneider released the SCD6000 RTU a while ago now, and while it has come with some good improvements, it has also come with a couple of shitty regressions. One of these shitty regressions is the use of the propriety Wind River C compiler for the compilation of SALL files. I'm guessing that this decision was made because Wind River makes VxWorks, which I believe is the operating system that runs on the SCD RTUs.  However because Wind River is based on GCC, and I don't want to pay many thousands of dollars for a compiler I only use a handful of times a year, I wondered if it was possible to use instead the very free GNU ARM embedded toolchain?

Now I don't actually have any SCD6000 hardware to test on, so this is just what I have come up with so far, and hopefully one day I'll get a chance to test it. On the other hand, if anyone out there wants to test this for me then please do so and let me know the results!

First, lets examine the process of how a SALL file is transformed into an elf binary.

  1. SALL file is converted into a C file.
    For SCD5200 RTUs this is done using the sall.exe file, located in C:\Program Files (x86)\Foxboro\RTU Station\utils\CalcsRTU_SALL\<firmware version>\buildsal\. You can run the program from the command line by using the following syntax: sall.exe infile.sal outfile.c errorfile.
    For SCD6000 RTUs this functionality is built into a dll file that is called directly from RTU Station.
  2. Compile and link the files into an elf file.
    For SCD5200 RTUs this is done with the Watcom compiler, and for SCD6000 RTUs this is done with the Wind River compiler. In both cases a make file is used to invoke the compiler and linker.

Read the rest of this entry »

09.27.17

Introducing protplot

As a amateur protection engineer, I am often frustrated with the protection grading tools that are out there, or more to the point aren't out there.  More often than not I end up using some clunky spreadsheet that does the job but it just doesn't look that pretty.

So I decided to do something about this situation, and have created protplot.  A simple grading tool that creates both a pretty graph, and a handy data table.

Example Plot

Example Plot

Read the rest of this entry »

| Posted in Software | No Comments »
09.25.17

The great wonderware bug hunt

I have ranted previously about a few bugs in System Platform 2017, but one in particular was quite interesting so I thought I would take the time to write it up as I learnt a few things along the way.

The Symptoms

After installing System Platform, other users that were not administrators, and not the use that installed System Platform could not log in.  Or rather they could log in, but the shell (explorer) would not load.  Some things that I tried:

  • Checked that all the shell entries were actually set to explorer.exe.  They were.
  • Checked that explorer was launching with procmon, which is was, but it quit early or got stuck without doing anything useful.  Couldn't see why.
  • Installing SP2017 on a clean, non domain joined machine.  Couldn't reproduce.
  • Insalling SP2017 on a clean, domain joined machine. Couldn't reproduce.

At this point I'm thinking one of our group policies was causing some kind of issue, but nothing was jumping out at me.  Around this time the GCS rep that I was in touch with about this bug got in touch and suggested it might be a permission issue, so I traced explorer.exe again, and filtered by ACCESS DENIED.  This time I got two or three entries that stood out.  So I took a look at them and compared them to another machine, and the obvious difference was that .\Users was missing from my SP2017 machine (and that aaUsers, aaPowerUsers, aaAdministrators also had permissions).  So it was now looking quite convincing that the System Platform installer was messing with the permissions, but why was it doing it on the machines in my domain, but not on other machines?

At this point I traced the entire installation with procmon, which generated some 40 million events and several GB of data.  After searching the data for permissions changes on the affected keys, I narrowed in on the program aahsecurity.exe.  I also enabled object auditing so that I could see exactly what permissions aahsecurity was changing and it clearly showed that the permissions on HKCR\CLSID\{xxx...} were having their inherited Users permissions removed, the question was... why?

HKCR

Something started to smell funny when I had a look at the permissions for HKCR\CLSID and noted that the .\Users didn't have permissions on this key, even though it should have been inheriting that permission from its parent HKCR.  Many will know that HKCR is actually a merging of HKLM\Software\Classes and HKCU\Software\Classes, so I began to wonder how the permissions are actually assigned when they could potentially be different from the different stores?  When comparing HKLM\Software\Classes\CLSID and HKCU\Software\Classes\CLSID  I noted that the HKCU not only existed where it didn't on the other machines I tested, but it didn't have the Users permission.  So it would appear that HKCR\CLSID has the permissions of the some franken combination of the HKLM and HKCU keys, but the subkeys still inherit the permissions from there physical parent.  This all seems quite logical.

So, aahsecurity was taking the permissions of HKCR\CLSID, adding aaUsers, aaPowerUsers and aaAdministrators and forcing inheritance down on all children which obliterates the .\Users permission previously inherited from HKLM\Software\Classes\CLSID.  What is should be doing, is applying these permissions to HKLM\Software\Classes\CLSID instead.  All of the above also applies to HKLM\Software\Classes\Wow6432Node\CLSID as well.

The fix is to delete the HKCU\Software\Classes\{Wow6432Node\}CLSID either before installation or if done after installation you will need to run aahsecurity again. It might be easiest to create a new user to do this because if your HKCU\Software\Classes\{Wow6432Node\}CLSID is not empty it will cause other problems.

However, one question remains:  aaUsers contains BUILTIN\Users.  So why don't the permissions propagate through the groups?  This is something I don't know and I have an open question on server fault if anyone knows why https://serverfault.com/questions/874319/registry-permissions-with-nested-groups

UPDATE: Wonderware has released a technote regarding this issue, and hotfix L00146083 is available.

 

| Posted in SCADA | No Comments »
09.15.17

Attempts to deploy System Platform 2017 into production

Well System Platform 2017 was released, and as I anticipated there are still lots of bugs in the software.  I have had to create three four several support requests with Schneider so far in an attempt to resolve various issues, many of which are individually totally blocking any sort of deployment.  This is on top of the issues I have had in obtaining the correct licenses - it is somewhat embarrassing having to explain to Schneider employees how their own licensing system works, and it is frustrating having to wait a week to get a response to an email - since my initial request for a quote it has taken three months to get the numbers I wanted.

The SRs I have raised so far (in case anyone else experiences the same problems)

  • SR 103146922 - Corrupt Graphics.  Graphics in a new wizard object get corrupted after making changes to unrelated attributes, but still appear correctly in window maker.
  • SR 43612418 -Engineering Units don't appear as set in template at runtime.
    Unfortunately this had to be closed as I couldn't reproduce it on a clean install.  However on the affected machine not even a complete reinstall of the software resolves the problem.
    UPDATE: This looks like it went away after the aahsecurity issues were resolves
  • SR 43612419 - Invalid MxPropertyLockedEnum value - After adding attributes to an wizard object via selecting a wizard option, the attributes have an invalid MxPropertyLockedEnum value in the object browser.  This one isn't a big deal, but is still frustrating.
    UPDATE: A hotfix is available for this.
  • SR103147165 - Following installation of WSP2017, users cannot login to windows.  After installing WSP2017, the only users able to log in are the Local Administrator account, the users that installed WSP2017, and the ArchestrA user.  All other users are greeted with a black screen (windows explorer silently exits on startup).
    UPDATE: I believe this is a bug in aahsecurity.exe that causes this the unusual circumstance where HKCU\Software\Classes\CLSID exists and does not have permissions assigned for BUILTIN\Users.  Workaround is either delete the key (only if it is empty!) before installing, or do it afterwards and run aahsecurity again. UPDATE2: Hotfix available
  • SR 103147203  - MxDataProvider service failure warning messages once a minute in the log, on a clean installation.
    UPDATE: This is related to the same permissions bug as above, same hotfix applies.
  • <SR Not Entered> - InTouch RT licenses do not work with InTouch View Apps, resulting in the error "This InTouch application could not acquire a ReadWrite license for Remote Desktop Services client".
  • <SR Not Entered> - Platform does not deploy, errors in the SMC log include "Could not Install MergeModuleMSI:DcomInstallPlatformMsi", "DeleteGlobalCacheSubFolders - failed" and "NamespaceMgr::LoadSecurityAssembly - LoadLibrary failed with Error 126".
    Fix: For me this issue was caused by LmxProxy not being able to be registered, which I discovered from the Windows Application log.  Copying LmxProxy.dll and LicAPINativeWrapper.dll from another machine to C:\Program Files (x86)\ArchestrA\Framework\Bin and running regsvr32 LmxProxy.dll seemed to fix this one.
  • <SR Not Entered> - After making changes to a platform, attempts to redeploy the platform fail, resulting in platform remaining shutdown and still in the changed state.
    Workaround: This seems to be a checkpointing issue.  Start the platform manually, then undeploy, then redeploy.

So my recommendation - test extensively before upgrading!

| Posted in SCADA | 2 Comments »
06.30.17

Repairing 'corrupt' ASE 1.x capture files

Recently I captured a whole bunch of data using an ASE V1 RTU test set to trouble shoot a comms issue, with the intention of analyzing the data once I got back to the office. Well when I did get back to the office and tried to open my files, ASE just sat there, blank, looking at me as if I were stupid. I only had one capture file out of a dozen that actually opened and it happened to be the smallest one. I figured that ASE had an issue opening large capture files, so I fired off an email to support asking for haaaaalp.

I was told that there was no issues viewing large files, but that I had likely following the incorrect procedure for capturing files which must be closed properly.  There was no hope of recovering my data.


Read the rest of this entry »

| Posted in Reversing | No Comments »
06.22.17

System Platform 2017 issues and workarounds

Well the final version has been released but there are still a few wrinkles - here are some workarounds that worked for me:

  • Graphic does not display in InTouch, with the following error in the log "Graphic element reference list incomplete or corrupted.  Failed to get embedded symbol at index <n>Last known symbol reference = <symbolname>"
    Fix: Don't know how to fix this one, working on it.
  • Some boolean attributes in object instances to not have the correct True/False labels, even though they are locked in the parent.
  • Fix: Export the object to .csv, delete the object, and reimport from csv.
  • An object does not remember the object wizard choices that you select after saving, they are always as they are assigned in the parent template.
    Fix: Unfortunately, the only way I have found to fix this is to export all of the objects of that template, and rebuild the template.
  • None of the object descriptions are present after importing objects from one galaxy to another (on a different machine), or if creating the galaxy from a CAB template.  I have experienced this importing objects between 2014R2 and 2017, and between two 2017 galaxies.  I have also heard of a case between two 2014R2 galaxies.
    Fix:  Create a galaxy backup of the original galaxy.  Create a galaxy with the same name on the other machine, and restore the galaxy backup using SMC.  Object descriptions should now be present.  Note that this will not work with different galaxy versions.
  • An object randomly shows in a bad state, and spits out various error messages, but commonly 'sets are not allowed on a quarantined object'.
    Fix: Dump the object to csv, delete, and reimport from csv.
  • After upgrading from 2014R2SP1, the FSGateway is still shown in the Operations Integration Server Manager.
    Fix: Find the value of HKCR\ArchestrA.FSGateway\CLSID, then delete the registry key HKCR\Wow6432Node\CLSID\{clsid-from-previous-step-here}

To summarize - suggest that one tests carefully before upgrading!  I have spent hours rebuilding broken objects and hunting bugs so far.  Lets hope the first bug fix release is not too far away.

| Posted in SCADA | No Comments »
04.6.17

Intouch Multimonitor - version two

Following on from my last post on the subject, I have now made my multimonitor application work for arbitrary screen layouts (but still limited to only two monitors).  Additionally, a number of subtle bug fixes have been implemented, such as the windows correctly showing following a NAD update.  Grab the updated package (for InTouch 2017) here.  To set the screen layout, simply select the picture that fits your situation on the settings page:

Multimonitor selection

Multimonitor selection

| Posted in SCADA | 4 Comments »