11.16.17

protplot update

Protplot has been updated with a couple of new features:

  • Improved UI - pretty icons, fixed several annoying bugs
  • Printing - can print to A3/A4
  • Added a pickup multiplier for inverse time curves
  • Added some more fuses curves
  • Now uses .Net Standard (for future windows store aspirations)
ProtPlot

ProtPlot

Source and binaries on github.

| Posted in Software | No Comments »
10.20.17

Running RTV and RTU Station on Windows 10

The SY-1101211D release of RTV and RTU Station does not support Windows 10,  which is no surprise since it took several years for Schneider to add support windows 7.   There are a couple of reasons that the standard installation doesn't work -

  • The included version of Java doesn't support Windows 10
  • Internal windows compatibility settings cause the RTU Station precheck process to elevate, which causes bash to crash for reasons I haven't bothered to investigate

To fix the first problem:

  1. Download the latest version of Java (8u151 at the time of writing) and install it - make sure you download the 32 bit version even if you are on a 64 bit system!
  2. Change the launcher.ini file in C:\Program Files (x86)\Foxboro\Remote Terminal Viewer, and edit the JRE line to JRE=C:\Program Files (x86)\Java\jre1.8.0_151\bin\javaw
  3. Open up the permissions for the C:\Program Files (x86)\Foxboro\Remote Terminal Viewer folder.  On my system the installer messed all the folder permissions up - we need the Users group to have Modify and Write permissions, so add those in if they don't exist.  If the splash screen never goes away after you open RTV, then these permissions are not correct.
  4. Done!  This is enough if you just have RTV installed.

If you have RTU station installed as well:

  1. In C:\Program Files (x86)\Foxboro\RTU Station\launcher.ini, change the JRE as above.
  2. Check the permissions for C:\Program Files (x86)\Foxboro\RTU Station\ as above.
  3. If after doing step 1 RTU Station always fails the precheck, try changing the PROGRAM line to
    PROGRAM=cmd /c "set __COMPAT_LAYER= & ..\cygwin\bin\bash.exe -c "cd scripts && ./precheck.sh > ../startup.log 2>&1" "
    

    This wraps up bash in a cmd instance, where we disable the compatibility environment variable that causes it to auto-elevate.

  4. If RTU Station still fails quietly (getting stuck on the splash screen), then change the line ARG3=-Xmx1024m to ARG3=-Xmx512m (credit to the chap from the 2017 user group conference for coming up with this one!)

A special note - the backup\restore utility seems to not support the compressed backup format so if your old data backups don't restore then try to unzip them manually first.

| Posted in Foxboro | No Comments »
10.20.17

compiling SALL calcs for SCD6000 without WindRiver

Schneider released the SCD6000 RTU a while ago now, and while it has come with some good improvements, it has also come with a couple of shitty regressions. One of these shitty regressions is the use of the propriety Wind River C compiler for the compilation of SALL files. I'm guessing that this decision was made because Wind River makes VxWorks, which I believe is the operating system that runs on the SCD RTUs.  However because Wind River is based on GCC, and I don't want to pay many thousands of dollars for a compiler I only use a handful of times a year, I wondered if it was possible to use instead the very free GNU ARM embedded toolchain?

Now I don't actually have any SCD6000 hardware to test on, so this is just what I have come up with so far, and hopefully one day I'll get a chance to test it. On the other hand, if anyone out there wants to test this for me then please do so and let me know the results!

First, lets examine the process of how a SALL file is transformed into an elf binary.

  1. SALL file is converted into a C file.
    For SCD5200 RTUs this is done using the sall.exe file, located in C:\Program Files (x86)\Foxboro\RTU Station\utils\CalcsRTU_SALL\<firmware version>\buildsal\. You can run the program from the command line by using the following syntax: sall.exe infile.sal outfile.c errorfile.
    For SCD6000 RTUs this functionality is built into a dll file that is called directly from RTU Station.
  2. Compile and link the files into an elf file.
    For SCD5200 RTUs this is done with the Watcom compiler, and for SCD6000 RTUs this is done with the Wind River compiler. In both cases a make file is used to invoke the compiler and linker.

Read the rest of this entry »

09.27.17

Introducing protplot

As a amateur protection engineer, I am often frustrated with the protection grading tools that are out there, or more to the point aren't out there.  More often than not I end up using some clunky spreadsheet that does the job but it just doesn't look that pretty.

So I decided to do something about this situation, and have created protplot.  A simple grading tool that creates both a pretty graph, and a handy data table.

Example Plot

Example Plot

Read the rest of this entry »

| Posted in Software | No Comments »
09.25.17

The great wonderware bug hunt

I have ranted previously about a few bugs in System Platform 2017, but one in particular was quite interesting so I thought I would take the time to write it up as I learnt a few things along the way.

The Symptoms

After installing System Platform, other users that were not administrators, and not the use that installed System Platform could not log in.  Or rather they could log in, but the shell (explorer) would not load.  Some things that I tried:

  • Checked that all the shell entries were actually set to explorer.exe.  They were.
  • Checked that explorer was launching with procmon, which is was, but it quit early or got stuck without doing anything useful.  Couldn't see why.
  • Installing SP2017 on a clean, non domain joined machine.  Couldn't reproduce.
  • Insalling SP2017 on a clean, domain joined machine. Couldn't reproduce.

At this point I'm thinking one of our group policies was causing some kind of issue, but nothing was jumping out at me.  Around this time the GCS rep that I was in touch with about this bug got in touch and suggested it might be a permission issue, so I traced explorer.exe again, and filtered by ACCESS DENIED.  This time I got two or three entries that stood out.  So I took a look at them and compared them to another machine, and the obvious difference was that .\Users was missing from my SP2017 machine (and that aaUsers, aaPowerUsers, aaAdministrators also had permissions).  So it was now looking quite convincing that the System Platform installer was messing with the permissions, but why was it doing it on the machines in my domain, but not on other machines?

At this point I traced the entire installation with procmon, which generated some 40 million events and several GB of data.  After searching the data for permissions changes on the affected keys, I narrowed in on the program aahsecurity.exe.  I also enabled object auditing so that I could see exactly what permissions aahsecurity was changing and it clearly showed that the permissions on HKCR\CLSID\{xxx...} were having their inherited Users permissions removed, the question was... why?

HKCR

Something started to smell funny when I had a look at the permissions for HKCR\CLSID and noted that the .\Users didn't have permissions on this key, even though it should have been inheriting that permission from its parent HKCR.  Many will know that HKCR is actually a merging of HKLM\Software\Classes and HKCU\Software\Classes, so I began to wonder how the permissions are actually assigned when they could potentially be different from the different stores?  When comparing HKLM\Software\Classes\CLSID and HKCU\Software\Classes\CLSID  I noted that the HKCU not only existed where it didn't on the other machines I tested, but it didn't have the Users permission.  So it would appear that HKCR\CLSID has the permissions of the some franken combination of the HKLM and HKCU keys, but the subkeys still inherit the permissions from there physical parent.  This all seems quite logical.

So, aahsecurity was taking the permissions of HKCR\CLSID, adding aaUsers, aaPowerUsers and aaAdministrators and forcing inheritance down on all children which obliterates the .\Users permission previously inherited from HKLM\Software\Classes\CLSID.  What is should be doing, is applying these permissions to HKLM\Software\Classes\CLSID instead.  All of the above also applies to HKLM\Software\Classes\Wow6432Node\CLSID as well.

The fix is to delete the HKCU\Software\Classes\{Wow6432Node\}CLSID either before installation or if done after installation you will need to run aahsecurity again. It might be easiest to create a new user to do this because if your HKCU\Software\Classes\{Wow6432Node\}CLSID is not empty it will cause other problems.

However, one question remains:  aaUsers contains BUILTIN\Users.  So why don't the permissions propagate through the groups?  This is something I don't know and I have an open question on server fault if anyone knows why https://serverfault.com/questions/874319/registry-permissions-with-nested-groups

UPDATE: Wonderware has released a technote regarding this issue, and hotfix L00146083 is available.

 

| Posted in SCADA | No Comments »
09.15.17

Attempts to deploy System Platform 2017 into production

Well System Platform 2017 was released, and as I anticipated there are still lots of bugs in the software.  I have had to create three four several support requests with Schneider so far in an attempt to resolve various issues, many of which are individually totally blocking any sort of deployment.  This is on top of the issues I have had in obtaining the correct licenses - it is somewhat embarrassing having to explain to Schneider employees how their own licensing system works, and it is frustrating having to wait a week to get a response to an email - since my initial request for a quote it has taken three months to get the numbers I wanted.

The SRs I have raised so far (in case anyone else experiences the same problems)

  • SR 103146922 - Corrupt Graphics.  Graphics in a new wizard object get corrupted after making changes to unrelated attributes, but still appear correctly in window maker.
  • SR 43612418 -Engineering Units don't appear as set in template at runtime.
    Unfortunately this had to be closed as I couldn't reproduce it on a clean install.  However on the affected machine not even a complete reinstall of the software resolves the problem.
    UPDATE: This looks like it went away after the aahsecurity issues were resolves
  • SR 43612419 - Invalid MxPropertyLockedEnum value - After adding attributes to an wizard object via selecting a wizard option, the attributes have an invalid MxPropertyLockedEnum value in the object browser.  This one isn't a big deal, but is still frustrating.
    UPDATE: A hotfix is available for this.
  • SR103147165 - Following installation of WSP2017, users cannot login to windows.  After installing WSP2017, the only users able to log in are the Local Administrator account, the users that installed WSP2017, and the ArchestrA user.  All other users are greeted with a black screen (windows explorer silently exits on startup).
    UPDATE: I believe this is a bug in aahsecurity.exe that causes this the unusual circumstance where HKCU\Software\Classes\CLSID exists and does not have permissions assigned for BUILTIN\Users.  Workaround is either delete the key (only if it is empty!) before installing, or do it afterwards and run aahsecurity again. UPDATE2: Hotfix available
  • SR 103147203  - MxDataProvider service failure warning messages once a minute in the log, on a clean installation.
    UPDATE: This is related to the same permissions bug as above, same hotfix applies.
  • <SR Not Entered> - InTouch RT licenses do not work with InTouch View Apps, resulting in the error "This InTouch application could not acquire a ReadWrite license for Remote Desktop Services client".
  • <SR Not Entered> - Platform does not deploy, errors in the SMC log include "Could not Install MergeModuleMSI:DcomInstallPlatformMsi", "DeleteGlobalCacheSubFolders - failed" and "NamespaceMgr::LoadSecurityAssembly - LoadLibrary failed with Error 126".
    Fix: For me this issue was caused by LmxProxy not being able to be registered, which I discovered from the Windows Application log.  Copying LmxProxy.dll and LicAPINativeWrapper.dll from another machine to C:\Program Files (x86)\ArchestrA\Framework\Bin and running regsvr32 LmxProxy.dll seemed to fix this one.
  • <SR Not Entered> - After making changes to a platform, attempts to redeploy the platform fail, resulting in platform remaining shutdown and still in the changed state.
    Workaround: This seems to be a checkpointing issue.  Start the platform manually, then undeploy, then redeploy.

So my recommendation - test extensively before upgrading!

| Posted in SCADA | 2 Comments »
06.30.17

Repairing 'corrupt' ASE 1.x capture files

Recently I captured a whole bunch of data using an ASE V1 RTU test set to trouble shoot a comms issue, with the intention of analyzing the data once I got back to the office. Well when I did get back to the office and tried to open my files, ASE just sat there, blank, looking at me as if I were stupid. I only had one capture file out of a dozen that actually opened and it happened to be the smallest one. I figured that ASE had an issue opening large capture files, so I fired off an email to support asking for haaaaalp.

I was told that there was no issues viewing large files, but that I had likely following the incorrect procedure for capturing files which must be closed properly.  There was no hope of recovering my data.


Read the rest of this entry »

| Posted in Reversing | No Comments »
06.22.17

System Platform 2017 issues and workarounds

Well the final version has been released but there are still a few wrinkles - here are some workarounds that worked for me:

  • Graphic does not display in InTouch, with the following error in the log "Graphic element reference list incomplete or corrupted.  Failed to get embedded symbol at index <n>Last known symbol reference = <symbolname>"
    Fix: Don't know how to fix this one, working on it.
  • Some boolean attributes in object instances to not have the correct True/False labels, even though they are locked in the parent.
  • Fix: Export the object to .csv, delete the object, and reimport from csv.
  • An object does not remember the object wizard choices that you select after saving, they are always as they are assigned in the parent template.
    Fix: Unfortunately, the only way I have found to fix this is to export all of the objects of that template, and rebuild the template.
  • None of the object descriptions are present after importing objects from one galaxy to another (on a different machine), or if creating the galaxy from a CAB template.  I have experienced this importing objects between 2014R2 and 2017, and between two 2017 galaxies.  I have also heard of a case between two 2014R2 galaxies.
    Fix:  Create a galaxy backup of the original galaxy.  Create a galaxy with the same name on the other machine, and restore the galaxy backup using SMC.  Object descriptions should now be present.  Note that this will not work with different galaxy versions.
  • An object randomly shows in a bad state, and spits out various error messages, but commonly 'sets are not allowed on a quarantined object'.
    Fix: Dump the object to csv, delete, and reimport from csv.
  • After upgrading from 2014R2SP1, the FSGateway is still shown in the Operations Integration Server Manager.
    Fix: Find the value of HKCR\ArchestrA.FSGateway\CLSID, then delete the registry key HKCR\Wow6432Node\CLSID\{clsid-from-previous-step-here}

To summarize - suggest that one tests carefully before upgrading!  I have spent hours rebuilding broken objects and hunting bugs so far.  Lets hope the first bug fix release is not too far away.

| Posted in SCADA | No Comments »
04.6.17

Intouch Multimonitor - version two

Following on from my last post on the subject, I have now made my multimonitor application work for arbitrary screen layouts (but still limited to only two monitors).  Additionally, a number of subtle bug fixes have been implemented, such as the windows correctly showing following a NAD update.  Grab the updated package (for InTouch 2017) here.  To set the screen layout, simply select the picture that fits your situation on the settings page:

Multimonitor selection

Multimonitor selection

| Posted in SCADA | 2 Comments »
02.21.17

System Platform 2017 (Omni) Thoughts

This has to be the biggest overhaul System Platform has seen in some time, and there is some awesome new stuff there.  Lets take a look at the major new features.

Wizard based object templates

This is the thing I am most excited about.  I have long wanted an equivalent of 'Interfaces' in a programming language in ArchestrA, and with the addition of 'choices' to the templating system in Omni this allows us to do basically the same thing.  I have been able to reduce the large amount of template sprawl that was occurring in 2014R2SP1 and previous and also reduce the memory footprint of objects by not having unused attributes present.  Of course there is some learning to do with this, but I found it all very natural and didn't need to consult the manual at all.

New ViewApp container

This looks like it is going in the right direction.  Flexible layouts, pan and zoom, resolution independence and lots of other good stuff.  I have only had a brief play with it, but it certainly looks like a good start although it has a long way to go before it will be ready for use in a complex production environment.

Historian security improvements

Lots of work done here to make sure that the sql accounts created for historian are no longer using default passwords, and the unnecessary creation of accounts is no longer.  Might seem like an unimportant detail, but I think it's great as it saves a lot of hassle afterwards if you are security conscious.

Preserve runtime changes

This is a super handy feature that allows the runtime changes to be preserved when deploying changes.  It means that you don't have to go through the hassle of writing values out to disk, or remembering to do an 'upload runtime changes' before doing a redeploy.

The Bad

Unfortunately there is some bad news as well.
The new graphics container does not look to me like it will be ready for production, except for the simplest of installations.

  • .NET controls do not work in the current build, and from what I've seen there isn't much priority going into fixing this.  This alone makes the new ViewApp container useless to many.
  • The new Apps (basically a turbo charged .Net control) only work in isolation.  You can't script them, and can't make them interact with anything unless it was part of the controls design.  This too is a deal breaker for many.
  • You can no longer have a modeless popup.  They have crudely sacrificed this function to show graphics in the new ViewApp container instead of creating a new animation for this purpose.
  • UPDATE: It turns out modal popup graphics don't link the symbol wizard options properly either just as they don't work in InTouch!
  • Changes made to linked graphics in templates do not propagate down to existing graphics.  You must reinsert these graphics for them to display correctly.
  • The map app does not support custom tile layers.  Allegedly base layers are supported but I couldn't get it to work.  Custom tile overlay layers are not supported, which is a major oversight in my opinion.

More bad news, is that you can't practically use the new object based templates with InTouch either, because any graphic associated with one of these object will not display correctly if it is directly embedded in InTouch, or shown as a popup.  They only show correctly when shown as a non popup graphic embedded in a parent symbol.  Wonderware have confirmed that this behaviour will not be fixed until a later release of System Platform.

Conclusion

I am sadly forced to conclude, that I will be unable to take advantage of any of the major new features in this release - the view app container isn't up to scratch yet, and the new wizard based objects do not show correctly in InTouch.  At least I can design all my new templates so that they will be ready to go when the time comes that I will be able to use them.

| Posted in SCADA | No Comments »