02.1.20

Guide to rolling your own OSI Monarch Extractor #2 - Decisions, Decisions

Now that you really know your system, you need to make some high level decisions about what you are going to model, and how you are going to do it.

Import Strategy

Your import strategy defines how you import data from GIS and other systems into the ADMS to form the model. You can perform three types of import: Full, Incremental, or Group. Full imports import the entire data model, and for small systems you this might be all that's required. Incremental imports take only the data that has been changed since the last import, and group imports only import the specific groups requested. You can use any combination of methods that suits. If using incremental imports, you need a method of generating incremental change sets from your data, and need to decide which data changes need to trigger inclusion in the change set. For our import strategy we used weekly full imports, and nightly incremental imports. Our implementation was driven by checkpoints in the GIS system, which means that data changes in the asset management database do not trigger changes in the ADMS. This was an acceptable trade off for us, but may not be for other implementations.

Next, you need to decide how to group devices. There are many ways to do this; an obvious choice is to group by feeder. In our implementation we decided that this was to large, and we instead have more granular grouping at the low voltage level. It helps if you can name your groups with a human readable identifier but this isn't always possible; in our implementation there are many small groups, so using internal ids was a more straightforward option.

Scope

The first thing to decide is what you want to model. Do you want to model your entire network, from source to load, or just the high voltage network, just your zone substations down, or something else? This decision may be driven by the quality of your data, or you business processes.

Sources

Sources are the things that energize your network. You should energize your network from points that have metering telemetry and per phase voltage telemetry, otherwise DPF may have trouble calculating flows between the source and the next downstream point that has metering telemetry.

Line impedances

Line impedances may be supplied in one of three formats:

  • Construction codes
  • Sequence components
  • Impedance matrix

These formats are described in detail in OSI documents, but you may wish to attempt to integrate the data from any existing modelling tools you might use already. For example in our implementation we converted existing data calculated with Leika into impedance matrices in IDF format, so we can be sure that our modelling tools (Sincal) and eMap are using the same line parameters.

AORs

AORs are a method of splitting up your network into areas that can have different access permissions. I'd suggest avoid using AOR 1 for eMap devices if possible, as this area is often used for system level alarms. Some system alarms have a setting to change the AOR, others do not. Since filtering alarms based on AOR can be quite useful, it helps to keep these alarms in a separate AOR from your eMap devices.

Once you have the high level design sorted, it's time to implement your extractor. But watch out for some traps... see the next post for details.

| Posted in SCADA | No Comments »
01.17.20

Guide to rolling your own OSI Monarch Extractor #1 - Know Thy System

The company I work for recently embarked on a journey to implement an Advanced Distribution Management System, and the solution we landed on was Open Systems International's Monarch platform. The core part of any ADMS is the network model which contains all the geographic and asset information you need to calculate real time connectivity and power flows for your electricity network. The Monarch platform is does not contain a GIS - it is designed to extract the model from your existing GIS and run alongside, and the software that performs this process is colloquially known as the Extractor. OSI provide extractors for a few mainstream GIS platforms, for example Esri and Intergraph. They do not provide one (at the time of writing at least) for GE's Smallworld, so we had to roll our own. This is the first in a series of posts that describes this journey for anyone else that happens to need to head down this path because there is no documentation provided to help you. Which brings me to the subject of this post: Know Thy System.

Read the rest of this entry »

| Posted in SCADA | No Comments »
06.16.18

Reverse engineering the innards of System Platform #1 - Packages, Templates, Objects, Primitives and Attributes

In order to try and understand various aspects of System Platform a little better, I have been poking around the inner workings of the software and discovering many interesting things, which I intend to share in a series of posts on the subject.  So here goes with the first one about Packages, Templates, Primitives and Attributes - settle in because it's a long one.

Firstly, we must must realise that the templates, primitives and attributes that we are going to talk about here are not the same as the templates and attributes that we are used to dealing with in the IDE which I will call IDE templates and IDE attributes from now on.

Templates are a base object type, such as $UserDefined, $Symbol, $DiscreteDevice and so on.  This list of templates is in the database in table template_definition, and for example mine looks like this:
Read the rest of this entry »

02.8.18

Top 8 improvements for System Platform/InTouch OMI

My top 8 improvements for System Platform (as at 2017 SP1):

  1. Overrides are not applied to object graphics, unless the graphic is embedded in a host graphic.  Unfortunately it doesn't seem to be a priority.  This means that if your object's graphics are altered by overrides, you can't display them directly in a OMI pane, popup window, or in the presenter app.  The only workaround is to create a symbol for each object graphic and embed the object graphic into it.
    UPDATE: This is apparently in the pipeline for whatever follows 2017 Update 2 (Update 3 presumably?)  Q4 2018
  2. The new alarm app doesn't have a column for the attribute description.  The alarm comment field does contain the description, but when you acknowledge the alarm it is overwritten either with a user comment or a fixed value.  In InTouch the alarm client can be scripted to keep the description in place, but there is no scripting ability in OMI.
    UPDATE: This is apparently fixed in 2017 Update 2
  3. When changing the navigation item in a graphic  script, it is done asynchronously.  This means that you cant change the navigation item programmatically then immediately show a graphic, because the autonavigation will kick in after your graphic is shown and close it again.
  4. Graphics shown on the map have no ability to be clicked on and navigate to a graphic in place of the map.  The only option to achieve this functionality is to use autonavigation which is not desirable in many cases.
    UPDATE: This is possibly fixed in 2017 Update 2...
  5. Windows that were opened in a multitabbed pane, are closed when the navigation object is changed.  There should be an option to keep these open.
  6. Multiple screens should have an option to have a separate navigation session for each screen.  The whole point of having multiple screens is that sometimes you want to see two different things at once.
    UPDATE: This has been noted and is being worked on for a future release
  7. IDE performance is terrible when checking in complex templates with many children.  One of my templates has 120 attributes, and 100 choices/options, and 100 children templates/instances.  It takes 10 minutes to check in which is painful.  It will be many times worse when I go to implement the rest of the 200 instances that I haven't done yet.  Complex templates also takes quite a while open in the IDE, but at least when it's open it is reasonably responsive.
  8. Sometimes data subscriptions get stuck, and are paused for about 30 seconds before completing.
    UPDATE: This can be fixed by deleting all your objects and importing them all again.  Hopefully this has been fixed in update 2...

| Posted in SCADA | 2 Comments »
02.1.18

The ArchestrA Attribute Wrangler

One frustration that I often have with ArchestrA is wanting to do bulk updates to attributes on one or more objects.  Things such as,

  • Updating a the description on an attribute that has a typo for all objects descending from a template
  • Change all the alarm priorities on objects hosted in a particular area
  • Changing the IO references for all objects using a particular topic name


Read the rest of this entry »

| Posted in SCADA, Software | 8 Comments »
11.22.17

System Platform 2017 Update 1

Here are some things that are still not fixed in Update 1:

  • Graphic overrides do not work in InTouch or OMI, unless they are pre-placed into a parent graphic.  This is my biggest pet peeve, I have 100s of symbols that are simply a container for an object graphic.  I honestly can't imagine what possible reason there is for deciding that this functionality is not important.
  • Graphics that are affected by wizard options do not propagate correctly in InTouch.  This functionality has not been implemented yet, and is of low importance apparently.
  • Exporting object to csv often results in errors on reimport.  Might have to put in a SR for this.

Here are some new things:

  • The SDK!  It's actually quite easy to use, although the samples that are distributed with the SDK might seem complicated if you aren't familiar with MVVM and WPF.  I have already ported the aaAttributeBrowser app to work with this.
  • Apps work much better.  Breadcrumb, Titlebar and Tree navigation apps in particular are quite a bit better than they were previously.
  • Lots of bugs fixed in the map app, but a still few present as well.

Some warnings about migrating an existing galaxy: when I migrated from 2017 to 2017 update 1, my objects got pretty messed up.  All my object instances gained all the attributes of their parent templates, even though most were disabled via wizard options.  When I exported all of my instances via galaxy dump and reimported, lots of them got imported with the wrong parent template which wasn't at all useful.  I ended up hosing all my objects and templates and importing them from my previous backup and things were well from there.

| Posted in SCADA | No Comments »
10.20.17

compiling SALL calcs for SCD6000 without WindRiver

Schneider released the SCD6000 RTU a while ago now, and while it has come with some good improvements, it has also come with a couple of shitty regressions. One of these shitty regressions is the use of the propriety Wind River C compiler for the compilation of SALL files. I'm guessing that this decision was made because Wind River makes VxWorks, which I believe is the operating system that runs on the SCD RTUs.  However because Wind River is based on GCC, and I don't want to pay many thousands of dollars for a compiler I only use a handful of times a year, I wondered if it was possible to use instead the very free GNU ARM embedded toolchain?

Now I don't actually have any SCD6000 hardware to test on, so this is just what I have come up with so far, and hopefully one day I'll get a chance to test it. On the other hand, if anyone out there wants to test this for me then please do so and let me know the results!

First, lets examine the process of how a SALL file is transformed into an elf binary.

  1. SALL file is converted into a C file.
    For SCD5200 RTUs this is done using the sall.exe file, located in C:\Program Files (x86)\Foxboro\RTU Station\utils\CalcsRTU_SALL\<firmware version>\buildsal\. You can run the program from the command line by using the following syntax: sall.exe infile.sal outfile.c errorfile.
    For SCD6000 RTUs this functionality is built into a dll file that is called directly from RTU Station.
  2. Compile and link the files into an elf file.
    For SCD5200 RTUs this is done with the Watcom compiler, and for SCD6000 RTUs this is done with the Wind River compiler. In both cases a make file is used to invoke the compiler and linker.

Read the rest of this entry »

09.25.17

The great wonderware bug hunt

I have ranted previously about a few bugs in System Platform 2017, but one in particular was quite interesting so I thought I would take the time to write it up as I learnt a few things along the way.

The Symptoms

After installing System Platform, other users that were not administrators, and not the use that installed System Platform could not log in.  Or rather they could log in, but the shell (explorer) would not load.  Some things that I tried:

  • Checked that all the shell entries were actually set to explorer.exe.  They were.
  • Checked that explorer was launching with procmon, which is was, but it quit early or got stuck without doing anything useful.  Couldn't see why.
  • Installing SP2017 on a clean, non domain joined machine.  Couldn't reproduce.
  • Insalling SP2017 on a clean, domain joined machine. Couldn't reproduce.

At this point I'm thinking one of our group policies was causing some kind of issue, but nothing was jumping out at me.  Around this time the GCS rep that I was in touch with about this bug got in touch and suggested it might be a permission issue, so I traced explorer.exe again, and filtered by ACCESS DENIED.  This time I got two or three entries that stood out.  So I took a look at them and compared them to another machine, and the obvious difference was that .\Users was missing from my SP2017 machine (and that aaUsers, aaPowerUsers, aaAdministrators also had permissions).  So it was now looking quite convincing that the System Platform installer was messing with the permissions, but why was it doing it on the machines in my domain, but not on other machines?

At this point I traced the entire installation with procmon, which generated some 40 million events and several GB of data.  After searching the data for permissions changes on the affected keys, I narrowed in on the program aahsecurity.exe.  I also enabled object auditing so that I could see exactly what permissions aahsecurity was changing and it clearly showed that the permissions on HKCR\CLSID\{xxx...} were having their inherited Users permissions removed, the question was... why?

HKCR

Something started to smell funny when I had a look at the permissions for HKCR\CLSID and noted that the .\Users didn't have permissions on this key, even though it should have been inheriting that permission from its parent HKCR.  Many will know that HKCR is actually a merging of HKLM\Software\Classes and HKCU\Software\Classes, so I began to wonder how the permissions are actually assigned when they could potentially be different from the different stores?  When comparing HKLM\Software\Classes\CLSID and HKCU\Software\Classes\CLSID  I noted that the HKCU not only existed where it didn't on the other machines I tested, but it didn't have the Users permission.  So it would appear that HKCR\CLSID has the permissions of the some franken combination of the HKLM and HKCU keys, but the subkeys still inherit the permissions from there physical parent.  This all seems quite logical.

So, aahsecurity was taking the permissions of HKCR\CLSID, adding aaUsers, aaPowerUsers and aaAdministrators and forcing inheritance down on all children which obliterates the .\Users permission previously inherited from HKLM\Software\Classes\CLSID.  What is should be doing, is applying these permissions to HKLM\Software\Classes\CLSID instead.  All of the above also applies to HKLM\Software\Classes\Wow6432Node\CLSID as well.

The fix is to delete the HKCU\Software\Classes\{Wow6432Node\}CLSID either before installation or if done after installation you will need to run aahsecurity again. It might be easiest to create a new user to do this because if your HKCU\Software\Classes\{Wow6432Node\}CLSID is not empty it will cause other problems.

However, one question remains:  aaUsers contains BUILTIN\Users.  So why don't the permissions propagate through the groups?  This is something I don't know and I have an open question on server fault if anyone knows why https://serverfault.com/questions/874319/registry-permissions-with-nested-groups

UPDATE: Wonderware has released a technote regarding this issue, and hotfix L00146083 is available.

 

| Posted in SCADA | No Comments »
09.15.17

Attempts to deploy System Platform 2017 into production

Well System Platform 2017 was released, and as I anticipated there are still lots of bugs in the software.  I have had to create three four several support requests with Schneider so far in an attempt to resolve various issues, many of which are individually totally blocking any sort of deployment.  This is on top of the issues I have had in obtaining the correct licenses - it is somewhat embarrassing having to explain to Schneider employees how their own licensing system works, and it is frustrating having to wait a week to get a response to an email - since my initial request for a quote it has taken three months to get the numbers I wanted.

The SRs I have raised so far (in case anyone else experiences the same problems)

  • SR 103146922 - Corrupt Graphics.  Graphics in a new wizard object get corrupted after making changes to unrelated attributes, but still appear correctly in window maker.
  • SR 43612418 -Engineering Units don't appear as set in template at runtime.
    Unfortunately this had to be closed as I couldn't reproduce it on a clean install.  However on the affected machine not even a complete reinstall of the software resolves the problem.
    UPDATE: This looks like it went away after the aahsecurity issues were resolves
  • SR 43612419 - Invalid MxPropertyLockedEnum value - After adding attributes to an wizard object via selecting a wizard option, the attributes have an invalid MxPropertyLockedEnum value in the object browser.  This one isn't a big deal, but is still frustrating.
    UPDATE: A hotfix is available for this.
  • SR103147165 - Following installation of WSP2017, users cannot login to windows.  After installing WSP2017, the only users able to log in are the Local Administrator account, the users that installed WSP2017, and the ArchestrA user.  All other users are greeted with a black screen (windows explorer silently exits on startup).
    UPDATE: I believe this is a bug in aahsecurity.exe that causes this the unusual circumstance where HKCU\Software\Classes\CLSID exists and does not have permissions assigned for BUILTIN\Users.  Workaround is either delete the key (only if it is empty!) before installing, or do it afterwards and run aahsecurity again. UPDATE2: Hotfix available
  • SR 103147203  - MxDataProvider service failure warning messages once a minute in the log, on a clean installation.
    UPDATE: This is related to the same permissions bug as above, same hotfix applies.
  • <SR Not Entered> - InTouch RT licenses do not work with InTouch View Apps, resulting in the error "This InTouch application could not acquire a ReadWrite license for Remote Desktop Services client".
  • <SR Not Entered> - Platform does not deploy, errors in the SMC log include "Could not Install MergeModuleMSI:DcomInstallPlatformMsi", "DeleteGlobalCacheSubFolders - failed" and "NamespaceMgr::LoadSecurityAssembly - LoadLibrary failed with Error 126".
    Fix: For me this issue was caused by LmxProxy not being able to be registered, which I discovered from the Windows Application log.  Copying LmxProxy.dll and LicAPINativeWrapper.dll from another machine to C:\Program Files (x86)\ArchestrA\Framework\Bin and running regsvr32 LmxProxy.dll seemed to fix this one.
  • <SR Not Entered> - After making changes to a platform, attempts to redeploy the platform fail, resulting in platform remaining shutdown and still in the changed state.
    Workaround: This seems to be a checkpointing issue.  Start the platform manually, then undeploy, then redeploy.

So my recommendation - test extensively before upgrading!

| Posted in SCADA | 2 Comments »
06.22.17

System Platform 2017 issues and workarounds

Well the final version has been released but there are still a few wrinkles - here are some workarounds that worked for me:

  • Graphic does not display in InTouch, with the following error in the log "Graphic element reference list incomplete or corrupted.  Failed to get embedded symbol at index <n>Last known symbol reference = <symbolname>"
    Fix: Don't know how to fix this one, working on it.
  • Some boolean attributes in object instances to not have the correct True/False labels, even though they are locked in the parent.
  • Fix: Export the object to .csv, delete the object, and reimport from csv.
  • An object does not remember the object wizard choices that you select after saving, they are always as they are assigned in the parent template.
    Fix: Unfortunately, the only way I have found to fix this is to export all of the objects of that template, and rebuild the template.
  • None of the object descriptions are present after importing objects from one galaxy to another (on a different machine), or if creating the galaxy from a CAB template.  I have experienced this importing objects between 2014R2 and 2017, and between two 2017 galaxies.  I have also heard of a case between two 2014R2 galaxies.
    Fix:  Create a galaxy backup of the original galaxy.  Create a galaxy with the same name on the other machine, and restore the galaxy backup using SMC.  Object descriptions should now be present.  Note that this will not work with different galaxy versions.
  • An object randomly shows in a bad state, and spits out various error messages, but commonly 'sets are not allowed on a quarantined object'.
    Fix: Dump the object to csv, delete, and reimport from csv.
  • After upgrading from 2014R2SP1, the FSGateway is still shown in the Operations Integration Server Manager.
    Fix: Find the value of HKCR\ArchestrA.FSGateway\CLSID, then delete the registry key HKCR\Wow6432Node\CLSID\{clsid-from-previous-step-here}

To summarize - suggest that one tests carefully before upgrading!  I have spent hours rebuilding broken objects and hunting bugs so far.  Lets hope the first bug fix release is not too far away.

| Posted in SCADA | No Comments »