06.16.18

Reverse engineering the innards of System Platform #1 - Packages, Templates, Objects, Primitives and Attributes

In order to try and understand various aspects of System Platform a little better, I have been poking around the inner workings of the software and discovering many interesting things, which I intend to share in a series of posts on the subject.  So here goes with the first one about Packages, Templates, Primitives and Attributes - settle in because it's a long one.

Firstly, we must must realise that the templates, primitives and attributes that we are going to talk about here are not the same as the templates and attributes that we are used to dealing with in the IDE which I will call IDE templates and IDE attributes from now on.

Templates are a base object type, such as $UserDefined, $Symbol, $DiscreteDevice and so on.  This list of templates is in the database in table template_definition, and for example mine looks like this:
Read the rest of this entry »

02.8.18

Top 8 improvements for System Platform/InTouch OMI

My top 8 improvements for System Platform (as at 2017 SP1):

  1. Overrides are not applied to object graphics, unless the graphic is embedded in a host graphic.  Unfortunately it doesn't seem to be a priority.  This means that if your object's graphics are altered by overrides, you can't display them directly in a OMI pane, popup window, or in the presenter app.  The only workaround is to create a symbol for each object graphic and embed the object graphic into it.
    UPDATE: This is apparently in the pipeline for whatever follows 2017 Update 2 (Update 3 presumably?)  Q4 2018
  2. The new alarm app doesn't have a column for the attribute description.  The alarm comment field does contain the description, but when you acknowledge the alarm it is overwritten either with a user comment or a fixed value.  In InTouch the alarm client can be scripted to keep the description in place, but there is no scripting ability in OMI.
    UPDATE: This is apparently fixed in 2017 Update 2
  3. When changing the navigation item in a graphic  script, it is done asynchronously.  This means that you cant change the navigation item programmatically then immediately show a graphic, because the autonavigation will kick in after your graphic is shown and close it again.
  4. Graphics shown on the map have no ability to be clicked on and navigate to a graphic in place of the map.  The only option to achieve this functionality is to use autonavigation which is not desirable in many cases.
    UPDATE: This is possibly fixed in 2017 Update 2...
  5. Windows that were opened in a multitabbed pane, are closed when the navigation object is changed.  There should be an option to keep these open.
  6. Multiple screens should have an option to have a separate navigation session for each screen.  The whole point of having multiple screens is that sometimes you want to see two different things at once.
    UPDATE: This has been noted and is being worked on for a future release
  7. IDE performance is terrible when checking in complex templates with many children.  One of my templates has 120 attributes, and 100 choices/options, and 100 children templates/instances.  It takes 10 minutes to check in which is painful.  It will be many times worse when I go to implement the rest of the 200 instances that I haven't done yet.  Complex templates also takes quite a while open in the IDE, but at least when it's open it is reasonably responsive.
  8. Sometimes data subscriptions get stuck, and are paused for about 30 seconds before completing.
    UPDATE: This can be fixed by deleting all your objects and importing them all again.  Hopefully this has been fixed in update 2...

| Posted in SCADA | No Comments »
02.1.18

The ArchestrA Attribute Wrangler

One frustration that I often have with ArchestrA is wanting to do bulk updates to attributes on one or more objects.  Things such as,

  • Updating a the description on an attribute that has a typo for all objects descending from a template
  • Change all the alarm priorities on objects hosted in a particular area
  • Changing the IO references for all objects using a particular topic name


Read the rest of this entry »

| Posted in SCADA, Software | No Comments »
11.22.17

System Platform 2017 Update 1

Here are some things that are still not fixed in Update 1:

  • Graphic overrides do not work in InTouch or OMI, unless they are pre-placed into a parent graphic.  This is my biggest pet peeve, I have 100s of symbols that are simply a container for an object graphic.  I honestly can't imagine what possible reason there is for deciding that this functionality is not important.
  • Graphics that are affected by wizard options do not propagate correctly in InTouch.  This functionality has not been implemented yet, and is of low importance apparently.
  • Exporting object to csv often results in errors on reimport.  Might have to put in a SR for this.

Here are some new things:

  • The SDK!  It's actually quite easy to use, although the samples that are distributed with the SDK might seem complicated if you aren't familiar with MVVM and WPF.  I have already ported the aaAttributeBrowser app to work with this.
  • Apps work much better.  Breadcrumb, Titlebar and Tree navigation apps in particular are quite a bit better than they were previously.
  • Lots of bugs fixed in the map app, but a still few present as well.

Some warnings about migrating an existing galaxy: when I migrated from 2017 to 2017 update 1, my objects got pretty messed up.  All my object instances gained all the attributes of their parent templates, even though most were disabled via wizard options.  When I exported all of my instances via galaxy dump and reimported, lots of them got imported with the wrong parent template which wasn't at all useful.  I ended up hosing all my objects and templates and importing them from my previous backup and things were well from there.

| Posted in SCADA | No Comments »
10.20.17

compiling SALL calcs for SCD6000 without WindRiver

Schneider released the SCD6000 RTU a while ago now, and while it has come with some good improvements, it has also come with a couple of shitty regressions. One of these shitty regressions is the use of the propriety Wind River C compiler for the compilation of SALL files. I'm guessing that this decision was made because Wind River makes VxWorks, which I believe is the operating system that runs on the SCD RTUs.  However because Wind River is based on GCC, and I don't want to pay many thousands of dollars for a compiler I only use a handful of times a year, I wondered if it was possible to use instead the very free GNU ARM embedded toolchain?

Now I don't actually have any SCD6000 hardware to test on, so this is just what I have come up with so far, and hopefully one day I'll get a chance to test it. On the other hand, if anyone out there wants to test this for me then please do so and let me know the results!

First, lets examine the process of how a SALL file is transformed into an elf binary.

  1. SALL file is converted into a C file.
    For SCD5200 RTUs this is done using the sall.exe file, located in C:\Program Files (x86)\Foxboro\RTU Station\utils\CalcsRTU_SALL\<firmware version>\buildsal\. You can run the program from the command line by using the following syntax: sall.exe infile.sal outfile.c errorfile.
    For SCD6000 RTUs this functionality is built into a dll file that is called directly from RTU Station.
  2. Compile and link the files into an elf file.
    For SCD5200 RTUs this is done with the Watcom compiler, and for SCD6000 RTUs this is done with the Wind River compiler. In both cases a make file is used to invoke the compiler and linker.

Read the rest of this entry »

09.25.17

The great wonderware bug hunt

I have ranted previously about a few bugs in System Platform 2017, but one in particular was quite interesting so I thought I would take the time to write it up as I learnt a few things along the way.

The Symptoms

After installing System Platform, other users that were not administrators, and not the use that installed System Platform could not log in.  Or rather they could log in, but the shell (explorer) would not load.  Some things that I tried:

  • Checked that all the shell entries were actually set to explorer.exe.  They were.
  • Checked that explorer was launching with procmon, which is was, but it quit early or got stuck without doing anything useful.  Couldn't see why.
  • Installing SP2017 on a clean, non domain joined machine.  Couldn't reproduce.
  • Insalling SP2017 on a clean, domain joined machine. Couldn't reproduce.

At this point I'm thinking one of our group policies was causing some kind of issue, but nothing was jumping out at me.  Around this time the GCS rep that I was in touch with about this bug got in touch and suggested it might be a permission issue, so I traced explorer.exe again, and filtered by ACCESS DENIED.  This time I got two or three entries that stood out.  So I took a look at them and compared them to another machine, and the obvious difference was that .\Users was missing from my SP2017 machine (and that aaUsers, aaPowerUsers, aaAdministrators also had permissions).  So it was now looking quite convincing that the System Platform installer was messing with the permissions, but why was it doing it on the machines in my domain, but not on other machines?

At this point I traced the entire installation with procmon, which generated some 40 million events and several GB of data.  After searching the data for permissions changes on the affected keys, I narrowed in on the program aahsecurity.exe.  I also enabled object auditing so that I could see exactly what permissions aahsecurity was changing and it clearly showed that the permissions on HKCR\CLSID\{xxx...} were having their inherited Users permissions removed, the question was... why?

HKCR

Something started to smell funny when I had a look at the permissions for HKCR\CLSID and noted that the .\Users didn't have permissions on this key, even though it should have been inheriting that permission from its parent HKCR.  Many will know that HKCR is actually a merging of HKLM\Software\Classes and HKCU\Software\Classes, so I began to wonder how the permissions are actually assigned when they could potentially be different from the different stores?  When comparing HKLM\Software\Classes\CLSID and HKCU\Software\Classes\CLSID  I noted that the HKCU not only existed where it didn't on the other machines I tested, but it didn't have the Users permission.  So it would appear that HKCR\CLSID has the permissions of the some franken combination of the HKLM and HKCU keys, but the subkeys still inherit the permissions from there physical parent.  This all seems quite logical.

So, aahsecurity was taking the permissions of HKCR\CLSID, adding aaUsers, aaPowerUsers and aaAdministrators and forcing inheritance down on all children which obliterates the .\Users permission previously inherited from HKLM\Software\Classes\CLSID.  What is should be doing, is applying these permissions to HKLM\Software\Classes\CLSID instead.  All of the above also applies to HKLM\Software\Classes\Wow6432Node\CLSID as well.

The fix is to delete the HKCU\Software\Classes\{Wow6432Node\}CLSID either before installation or if done after installation you will need to run aahsecurity again. It might be easiest to create a new user to do this because if your HKCU\Software\Classes\{Wow6432Node\}CLSID is not empty it will cause other problems.

However, one question remains:  aaUsers contains BUILTIN\Users.  So why don't the permissions propagate through the groups?  This is something I don't know and I have an open question on server fault if anyone knows why https://serverfault.com/questions/874319/registry-permissions-with-nested-groups

UPDATE: Wonderware has released a technote regarding this issue, and hotfix L00146083 is available.

 

| Posted in SCADA | No Comments »
09.15.17

Attempts to deploy System Platform 2017 into production

Well System Platform 2017 was released, and as I anticipated there are still lots of bugs in the software.  I have had to create three four several support requests with Schneider so far in an attempt to resolve various issues, many of which are individually totally blocking any sort of deployment.  This is on top of the issues I have had in obtaining the correct licenses - it is somewhat embarrassing having to explain to Schneider employees how their own licensing system works, and it is frustrating having to wait a week to get a response to an email - since my initial request for a quote it has taken three months to get the numbers I wanted.

The SRs I have raised so far (in case anyone else experiences the same problems)

  • SR 103146922 - Corrupt Graphics.  Graphics in a new wizard object get corrupted after making changes to unrelated attributes, but still appear correctly in window maker.
  • SR 43612418 -Engineering Units don't appear as set in template at runtime.
    Unfortunately this had to be closed as I couldn't reproduce it on a clean install.  However on the affected machine not even a complete reinstall of the software resolves the problem.
    UPDATE: This looks like it went away after the aahsecurity issues were resolves
  • SR 43612419 - Invalid MxPropertyLockedEnum value - After adding attributes to an wizard object via selecting a wizard option, the attributes have an invalid MxPropertyLockedEnum value in the object browser.  This one isn't a big deal, but is still frustrating.
    UPDATE: A hotfix is available for this.
  • SR103147165 - Following installation of WSP2017, users cannot login to windows.  After installing WSP2017, the only users able to log in are the Local Administrator account, the users that installed WSP2017, and the ArchestrA user.  All other users are greeted with a black screen (windows explorer silently exits on startup).
    UPDATE: I believe this is a bug in aahsecurity.exe that causes this the unusual circumstance where HKCU\Software\Classes\CLSID exists and does not have permissions assigned for BUILTIN\Users.  Workaround is either delete the key (only if it is empty!) before installing, or do it afterwards and run aahsecurity again. UPDATE2: Hotfix available
  • SR 103147203  - MxDataProvider service failure warning messages once a minute in the log, on a clean installation.
    UPDATE: This is related to the same permissions bug as above, same hotfix applies.
  • <SR Not Entered> - InTouch RT licenses do not work with InTouch View Apps, resulting in the error "This InTouch application could not acquire a ReadWrite license for Remote Desktop Services client".
  • <SR Not Entered> - Platform does not deploy, errors in the SMC log include "Could not Install MergeModuleMSI:DcomInstallPlatformMsi", "DeleteGlobalCacheSubFolders - failed" and "NamespaceMgr::LoadSecurityAssembly - LoadLibrary failed with Error 126".
    Fix: For me this issue was caused by LmxProxy not being able to be registered, which I discovered from the Windows Application log.  Copying LmxProxy.dll and LicAPINativeWrapper.dll from another machine to C:\Program Files (x86)\ArchestrA\Framework\Bin and running regsvr32 LmxProxy.dll seemed to fix this one.
  • <SR Not Entered> - After making changes to a platform, attempts to redeploy the platform fail, resulting in platform remaining shutdown and still in the changed state.
    Workaround: This seems to be a checkpointing issue.  Start the platform manually, then undeploy, then redeploy.

So my recommendation - test extensively before upgrading!

| Posted in SCADA | 2 Comments »
06.22.17

System Platform 2017 issues and workarounds

Well the final version has been released but there are still a few wrinkles - here are some workarounds that worked for me:

  • Graphic does not display in InTouch, with the following error in the log "Graphic element reference list incomplete or corrupted.  Failed to get embedded symbol at index <n>Last known symbol reference = <symbolname>"
    Fix: Don't know how to fix this one, working on it.
  • Some boolean attributes in object instances to not have the correct True/False labels, even though they are locked in the parent.
  • Fix: Export the object to .csv, delete the object, and reimport from csv.
  • An object does not remember the object wizard choices that you select after saving, they are always as they are assigned in the parent template.
    Fix: Unfortunately, the only way I have found to fix this is to export all of the objects of that template, and rebuild the template.
  • None of the object descriptions are present after importing objects from one galaxy to another (on a different machine), or if creating the galaxy from a CAB template.  I have experienced this importing objects between 2014R2 and 2017, and between two 2017 galaxies.  I have also heard of a case between two 2014R2 galaxies.
    Fix:  Create a galaxy backup of the original galaxy.  Create a galaxy with the same name on the other machine, and restore the galaxy backup using SMC.  Object descriptions should now be present.  Note that this will not work with different galaxy versions.
  • An object randomly shows in a bad state, and spits out various error messages, but commonly 'sets are not allowed on a quarantined object'.
    Fix: Dump the object to csv, delete, and reimport from csv.
  • After upgrading from 2014R2SP1, the FSGateway is still shown in the Operations Integration Server Manager.
    Fix: Find the value of HKCR\ArchestrA.FSGateway\CLSID, then delete the registry key HKCR\Wow6432Node\CLSID\{clsid-from-previous-step-here}

To summarize - suggest that one tests carefully before upgrading!  I have spent hours rebuilding broken objects and hunting bugs so far.  Lets hope the first bug fix release is not too far away.

| Posted in SCADA | No Comments »
04.6.17

Intouch Multimonitor - version two

Following on from my last post on the subject, I have now made my multimonitor application work for arbitrary screen layouts (but still limited to only two monitors).  Additionally, a number of subtle bug fixes have been implemented, such as the windows correctly showing following a NAD update.  Grab the updated package (for InTouch 2017) here.  To set the screen layout, simply select the picture that fits your situation on the settings page:

Multimonitor selection

Multimonitor selection

| Posted in SCADA | 2 Comments »
02.21.17

System Platform 2017 (Omni) Thoughts

This has to be the biggest overhaul System Platform has seen in some time, and there is some awesome new stuff there.  Lets take a look at the major new features.

Wizard based object templates

This is the thing I am most excited about.  I have long wanted an equivalent of 'Interfaces' in a programming language in ArchestrA, and with the addition of 'choices' to the templating system in Omni this allows us to do basically the same thing.  I have been able to reduce the large amount of template sprawl that was occurring in 2014R2SP1 and previous and also reduce the memory footprint of objects by not having unused attributes present.  Of course there is some learning to do with this, but I found it all very natural and didn't need to consult the manual at all.

New ViewApp container

This looks like it is going in the right direction.  Flexible layouts, pan and zoom, resolution independence and lots of other good stuff.  I have only had a brief play with it, but it certainly looks like a good start although it has a long way to go before it will be ready for use in a complex production environment.

Historian security improvements

Lots of work done here to make sure that the sql accounts created for historian are no longer using default passwords, and the unnecessary creation of accounts is no longer.  Might seem like an unimportant detail, but I think it's great as it saves a lot of hassle afterwards if you are security conscious.

Preserve runtime changes

This is a super handy feature that allows the runtime changes to be preserved when deploying changes.  It means that you don't have to go through the hassle of writing values out to disk, or remembering to do an 'upload runtime changes' before doing a redeploy.

The Bad

Unfortunately there is some bad news as well.
The new graphics container does not look to me like it will be ready for production, except for the simplest of installations.

  • .NET controls do not work in the current build, and from what I've seen there isn't much priority going into fixing this.  This alone makes the new ViewApp container useless to many.
  • The new Apps (basically a turbo charged .Net control) only work in isolation.  You can't script them, and can't make them interact with anything unless it was part of the controls design.  This too is a deal breaker for many.
  • You can no longer have a modeless popup.  They have crudely sacrificed this function to show graphics in the new ViewApp container instead of creating a new animation for this purpose.
  • UPDATE: It turns out modal popup graphics don't link the symbol wizard options properly either just as they don't work in InTouch!
  • Changes made to linked graphics in templates do not propagate down to existing graphics.  You must reinsert these graphics for them to display correctly.
  • The map app does not support custom tile layers.  Allegedly base layers are supported but I couldn't get it to work.  Custom tile overlay layers are not supported, which is a major oversight in my opinion.

More bad news, is that you can't practically use the new object based templates with InTouch either, because any graphic associated with one of these object will not display correctly if it is directly embedded in InTouch, or shown as a popup.  They only show correctly when shown as a non popup graphic embedded in a parent symbol.  Wonderware have confirmed that this behaviour will not be fixed until a later release of System Platform.

Conclusion

I am sadly forced to conclude, that I will be unable to take advantage of any of the major new features in this release - the view app container isn't up to scratch yet, and the new wizard based objects do not show correctly in InTouch.  At least I can design all my new templates so that they will be ready to go when the time comes that I will be able to use them.

| Posted in SCADA | No Comments »