The ArchestrAnaut

Made it back safely from Nashville, although there were a few minutes at the taping of the CMA Christmas Show I thought we were about the throw down.. but that’s a different story for another day.  I’m planning on writing up a few posts on what I enjoyed about the show, where I think they could improve, and most importantly some interesting tidbits I picked up.  There are some pretty juicy items, if you’re into things like that.

- Andy

In case you missed it.. I know I did, here’s a press release on the official launch of System Platform 2012

http://iom.invensys.com/EN/Pages/IOM_NewsDetail.aspx?NewsID=393

I have had access to the beta for a while and there are some really cool things in this release.  For fear of saying more than I should I’ll keep my trap shut until after the conference before I discuss any of the new features.

A couple items I will comment on, since they are in the press release, are the undying infatuation with Hyper-V and some under the hood improvements. 

Once again I think Invensys is showing their colors in terms of being squarely aligned with Microsoft.  I’m not saying this is a good or bad thing, just an observation.  Let’s just say in the doc pack with the Beta there is a massive document on deploying a more resilient system with Hyper-V.  I truly do wish they would take a more even handed approach and pay heed to the acknowledged 900 lb. gorilla in the space, VMWare.  Over the last 2 or 3 years while Hyper-V has caught up in a few areas, VMWare has zoomed ahead on countless fronts that Hyper-V hasn’t even thought about tackling.  I guess I just feel like they are doing a dis-service to the customer base by continuing to pound the drum for one hypervisor over another, especially when that hypervisor isn’t even close to best in class.  Ok, off my soapbox or I’ll tick off the wrong people again.

As for the good stuff, from what I’ve seen and read there are some serious under the hood improvements that I think everyone is going to really like.  Think reliability.  Once an engine and objects are up and running IAS is pretty durn good.  However, when’s the last time you felt good about deploying thousands of objects or an entire platform?  Based on some serious changes these scenarios should be much much better.  There aren’t a ton of new features on the surface, at least from a pure Archestra perspective, but one in particular should really excite folks in the qualified industries (i.e. Life Sciences).  This is an issue I brought to their attention a couple years ago at Ops Manage.  Good to see that real feedback from real customers eventually makes it into the product.

Ok, I’m starting to say too much now so I’ll let the proper Invensys people introduce the product at Ops Manage then I’ll do some write ups after that.

- Andy

Only a few more days to Ops Manage.  I’ve been trolling through the program to figure out what sessions I’ll be attending and what vendors I want to try to meet with.

I’m still planning on trying to hold an aaBeers get together.  Just a casual get together of fellow archestranauts to have a few beers and talk about whatever comes up.  What could be cooler than a bunch of geeks with nametags chatting about the merits of the MX protocol or beings IOPS bound on your checkpointing processes!!

I’m not much of a twitter master but I plan to try tweeting throughout the conference.  You can follow me at @archestranaut

http://twitter.com/#!/archestranaut

When I figure out a place for aaBeers I’ll probably tweet it along with a blog post.

Looking forward to seeing everyone in Nashvegas!

- Andy

First off, yes we’re still around.  Got tied up in a rush job on a DeltaV Batch project that has taken me away for the last 3 weeks or so. 

As promised in our last article we’re going to discuss how you can write a long running script without using asynchronous scripting.  To be totally fair this only works in the situation where your script is long running because you are iterating some kind of array.  If you need an async script because you have a really long database call or some long running file transaction there’s not much I can do for you.

Before we get into the technique the first thing you need to understand is what an asynchronous script is.  When IAS executes its objects, on a particular engine all objects and scripts in those objects are executed one after another.  It’s pretty amazing that the software engines can get through so many pieces of code so fast.  Kudos to the software architects.  If you find yourself with scan overruns on an engine, the first thing to do if figure out if you have any objects hanging or hogging resources.  If everything checks out there you should create a new engine and move some objects over.  At this point you have now split your object list and allowed them to run in parallel.  A while back I saw some advice that said you shouldn’t have more engines than cores on your computer.  That seems to make sense if we’re trying to run all the engines in parallel as efficiently as possible.  By the way… getting back to the previous post… does this help you understand why it takes at least one scan to hookup to an attribute that is on another engine?  Back to what we were talking about, there’s not a lot of good information in the books on what async scripts are but here’s one line I found that helps a little bit

Asynchronous scripting mode is a group of scripts running on the same, lower priority execution thread.

There are two key points here to think about.  First, the scripts are running on a different thread.  If you know much about software development you know that in a single thread all actions are lined up and happen one after another.  In other words, if there is a slow poke in the group then everybody waits behind them to get done before they can execute, kinda like the old person in the grocery store who still insists on writing a check.  Extending the analogy a little bit imagine you’re at a Wal-Mart supercenter with 32 checkout lines.  Only problem is the manager only has 3 lines open.  Wow, did I just tie-in Wal-Mart to a discussion of asynchronous scripting in Archestra?  The second part of the analogy makes us think about the difference in speed of the checkout folks. We all know that all cashiers are not created equal.  Imagine you have one cashier that is totally focused on the customer in front of them, that customer is a high priority.  Chances are the line is going to move really fast  Let’s call this cashier Fay.  Now imagine another cashier who puts a low priority on the customer in front of them, say his name is Cletus.  They are answering their cell phone, chatting with the guy stocking the drinks, commenting on what the person behind you is wearing, in general they are being interrupted by higher priority tasks.  That process is going to move pretty slowly.  When you run your scripts in a normal fashion you get into Fay’s line.  While the line may be long she’s really efficient and gets everyone through in a quick manner.  When you run your scripts async you are in Cletus’s line.  You’ll get through, eventually, but you may be waiting around for a while.  What I have left out of this discussion though, is that Fay can only work for 15 minutes at a time and if you are standing in line when her 15 minutes is up, then you have to put everything back and start over again… this is like a scan overrun.  Enough of the Wal-Mart analogies, back to the highbrow stuff.

So we have an array we’re trying to loop through but because of the length of the array and the nature of what we’re doing in the array, there is no way we can make it through the whole array in one scan.   How do you handle this.

Well, the first thing to do is figure out how long you have to execute your script.  This is available to you via some parameters on your engine.  Here’s the expression we use to calculate what we call a Halt Time.

Me._HaltTime = MyEngine.Scheduler.ScanTime + Me._ScanFraction * MyEngine.Scheduler.ScanPeriod / 1000;

Basically what we’re doing is saying take the start time of the current scan cycle then add a portion of the scheduled scan time.  We will typically use something like 75%.  Say we start this scan at 13.2 seconds after the minute.  If we have a 1 second scan time and a 75% scan fraction we end up calculating a halt time of 13.95 seconds after the minute.  We’ll use this later in our loop.

Next, we need to setup our tracking devices.  On the first time through our script we reset our tracking variable; say we call it j to 1.  You should keep track of what you have or haven’t done either via flags or some internal step counter.  Next, before you start looping, set your loop index, say we call it i , to whatever the value of j is.  What this is doing is allowing you to restart in the middle of your loop.  After each step you should be incrementing i along with setting j equal to i.  This is a bit hard to explain in prose so here it is in pseudocode.  Apologies in advance for my code not being as pretty as David’s.  I don’t have the patience to do all the formatting like he does.

Dim j as Integer (in declarations section of script so it persists across scans)
Dim i as Integer (in regular header part of script)

If (we’re just starting this sequence) Then
       j = 1
     Go to Step 2
End If

If (we’re at Step 2)
     ‘ Initialize the loop index counter to whatever the last value of j was.  First time through this is 1, next time through it might be somewhere between 1 and the end of the array
     i = j

     ‘ Start looping and bail out of the loop if we are at the end OR we have reached our pre-calculated halt time
     While (I < ArrayLength AND Now() < Halt Time)
                { Do yo thang….}
                 {Maybe a message saying you are x% done}
                 ‘Advance the counter and track the counter with j, which persists across scans
                 i = i + 1
                 j = i

      End While

     If (I >= ArrayLength) Then
              ‘ We’ve made it all the way through so reset j and advance to the next step
               j = 1
               Go to Step 3
      End If

So, if you follow what we’re doing we’re just using a secondary variable to keep up with how far we’ve made it in the array before we had to bail out.  Then, when we jump back in on the next scan we pick up where we left off.  Pretty slick huh?

One of the interesting things we discovered while doing our testing was the idea that Async scripts are definitely slower.  During a test using this particular method above we saw a distinct 5x slowdown (70 ms to 370 ms) when we swapped to Async from Sync,  changing nothing else.  That should give you some perspective on what it means to swap to a low priority thread.

I hope this was useful and worth the wait.

Anyone else have a method they’ve developed that accomplishes something similar?

Ok so that’s a really dramatic title for what we think is a pretty cool technique we’ve been working on for the last few months. 

A little back-story if you will.  We’re working on an application for a customer that requires us to download recipe parameters to a PLC.  No big deal, right?  Just use the FMO from the EOM.  First problem was that due to some licensing changes, using the FMO wasn’t an option.  Ok, well I guess we can just write our own FMO, that can’t be too hard.  We started down this path but quickly realized that having to create multiple UDA’s for every attribute was a maintenance nightmare and not something we wanted our customers to deal with.  By the way, to get a sense of scale for what this customer needed, we have a machine where we are downloading over 2000 recipe values to a single machine.  Yes, that 2 THOUSAND.  The machine isn’t that big but it does have a massive array of implements that all have about 16 set points so you have a matrix of about 16 * 100 for one section of the machine.. that’s how you get into thousands of formula parameters.

After batting around a number of ideas we came across what ended up being the ultimate solution, and that’s what we’re going to share with you.  We’re not going to do a massive code dump and give away the keys to the kingdom (there’s lots of hours in this method) but we will show you the main idea so you can possibly implement your own solution.

It all starts with Indirects.  If you’ve never used Indirects you really should learn about them.  Basically an Indirect is a variable type you can declare in a script.  Once you declare an indirect you call the “BindTo” function on that variable.  Once you have called the BindTo you can now read and write to that indirect and the values will through to the variable you have bound to.  This method can solve a ton of issues especially if you have an unknown quantity in an array or want to write a generic object that can work in a bunch of different settings.  Below is a really really simple example.

Dim X as Indirect;
Dim Timer as Integer;

X.BindTo(“AttributeName.Value”);

‘ Wait till the quality becomes good before attempting to read or write
While NotGood(X)
     Timer = Timer + 1;
EndWhile;

‘ Read from X
LogMessage(“Value of X is “ + X);

’ Write to X
X = 12.3;

There is some basic reading in the help files that give you a little more information.  Here’s where the magic starts. Way back in the 2.1 days there used to be a note… at least I think there was… that said you couldn’t bind across engines.  I always wondered why that was?  Turns out through some other investigations that hooking up to I/O that isn’t on the same engine takes at least one scan.  Why does that matter?  Well, if in a script you declare the Indirect at the top of the script that reference is destroyed at the end of the scan cycle.  If its’ destroyed at the end of the scan cycle then how can it persist across scans to survive to hook up to another engine?

So how do you get around this limitation?  Simple, move your declaration up into the rarely used declarations section of your script. 

If you didn’t know, variables declared in this section are persisted across scans.  Now we can declare an indirect in this section, do some binding in the script itself, and use the results of that binding in a subsequent scan.  Very cool!  I’ve been doing IAS for a long time and I was frankly amazed that I never put this little trick together (thanks David!).  Actually after using this for this technique I’ve started using these guys a lot more in place of placeholder UDA’s for basic things like flags and multi-scan counters.  Just remember, these are not checkpointed so anything that needs to persist across a failover should be a UDA, not one of these variables.

So now that we’ve got the dynamic part, what’s this hyper-scale thing?

So back to the formulas.  Since the FMO was out we basically wrote our own Formula database in SQL Server.  Not terribly difficult to do a basic one.  We ended up blinging this one out with lots of really cool features; that part we’re not going to share.  At the end of the day we make a stored procedure call to the database and it not only retrieves the basic parts of the recipe (names, values, units, limits, etc.) but it also downloads I/O references.  Using these I/O references we iterate over some arrays and dynamically bind our Indirects to the appropriate I/O source.  Did you notice our arrays of Indirects in our declarations?  Pretty sweet huh?

So what does all this mean?  This means that you can have an arbitrary set of parameters that you need to read and write to and as long as you can get a list of I/O addresses you can scale to your heart’s content.

One question you might ask is how do you see the values if they aren’t UDA’s?   We solve that issue by copying out the contents to an array of strings that is dynamically scaled based on the number of I/O we’re dealing with.  Super simple example below.

Dim Count as Integer;
Dim Index as Integer;

Count = Me.ValsArray.Dimension1;

For Index = 1 to Count

              Me.ValsArray[Index] = IndirectValsArray[Index];

Next Index;

Knowing that you can access the size of an array at runtime via the Dimension1 parameter makes things a lot more flexible.  Also, did you know that you can set the Dimension1 at runtime to dynamically change the size of the array?  Even cooler is that if you set Dimension1 to 0 then immediately to some positive value you just cleared out the array.  Is this more efficient than setting the size then iterating and clearing… don’t know, never tested that.

Back to the problem at hand.

At the end of the day what matters just as much as flexibility is how well it actually scales.  We’ve tested our solution at 10,000 I/O to a Control Logix L62 PLC (yes, it’s a big array so lots of efficiency there) and we run at about 250 ms to get through the entire array reading every attribute’s value and quality and copying them into our UDA arrays so we can view them with object viewer.  If you’re doing something with 10,000 I/O you might just have to accept a 250ms cycle time for something this far off the reserve.   With something this big I’m thinking you might be on your own engine to avoid async issues with other objects and scripts.  Something we have noticed in our testing from 1 element through 10,000 elements is that the scaling is pretty linear.  That’s good to know if we needed to keep going.  To be fair things may fall apart at 11,000 elements but it looks pretty good up to that 10,000.

If you were really astute, and you’ve worked with Indirects before, one question you might ask is how in the world you design a script that waits for all of the qualities to go good before reading and writing data.  That’s going to be the subject of our next post, designing a system that allows for multi-scan execution of a script without using Asynchronous scripts.  I know you’ll be waiting with baited breath for that one

- Andy 

A big red X for the Archestranaut.  I submitted a talk that would review and discuss the 3 or 4 most popular posts from the year.  Apparently it didn’t catch the right folks fancy.  That’s ok.  I guess one way to look at it is that I get to enjoy more of the show than I have in years past where I was a bundle of nerves before my talk.

With that being said I’m rolling around a couple ideas around an Archestranaut themed meetup.  In the Virtualization and Storage world a lot of people have started meetups called vBeers or StorageBeers.   Basically it’s nothing more than an excuse for some like minded geeks to get together and share a few beers over some technical.. or non-technical discussion.  If I can get a small critical mass of folks interested I’ll work on logistics, nothing too complicated.  If you might be interested in getting together some time during the conference to enjoy a few beers and chat with likeminded folks either leave a comment or shoot me an email directly.  Once I have enough people I’ll post info on whatever details I come up with.  Oh and one other thing.   This isn’t a fancy sponsored event where some vendor is paying for it all, just some regular guys and gals getting together.

- Andy

Here are a few posts I’ve been reading lately and you might be interested in.

Our take on what you can do to protect your ICS from the next cyber security threat. 

Stuxnet and What it Means to Our Security

One of our business development managers has started a new blog entitled Automate-Zen.  He’s a sales guy so you’re not going to get really deep technical deep dives.  What you will get is a little different take on what he’s seeing in the industry and specifically with our customers.

http://www.avidsolutionsinc.com/blog/automatezen/

Finally, Jay David has been really busy over at AutomationSuccess.com.  When I poked him a while back that we don’t see enough System Platform success profiles, at least I hadn’t found them, his response was essentially… “just wait and see”.  He wasn’t kidding.  If you have already been then it’s worth a return trip.  if you haven’t seen the blog yet, do yourself a favor and check it out.

http://www.automationsuccess.com/

- Andy

Seeing all the activity at VMWorld via the Twittersphere got me to thinking about some current technologies and how what we do in the “Galaxy” might carry over.

Then it hit me, Archestra is a lot like the famed “Cloud”.  Don’t get me wrong I’m not speaking of the Cloud in a condescending manner.  I think the technologies that make up the collective Cloud are set to transform the way we do work on a scale similar to the integration of the Internet into our daily work flow.  Answer this question honestly… do you feel stranded now when you have to work on your computer without an Internet connection… kinda like leaving the house to drive more than 5 miles away without a cell phone.  Oh the horror.  What will I do if I don’t see that email 1 minute after it’s sent.  I don’t think it’s too far fetched to think that a handful of years in the future we won’t be installing local applications for editing and composition.  Note that I am intentionally leaving out runtime applications as I think that’s a tougher sell to move off site.

Anyway, back to my original thought, how is Archestra like the Cloud and what can we learn from these similarities?

Read the rest of this entry »

I’ve been going to Ops Manage for a few years now and I can honestly say each year it has gotten progressively better.  I hear from the old timers about how over the top WonderWorld used to be but overall I think it’s a pretty good conference.  If you’re trying to convince yourself or your boss that you should be going here are some points to consider and things to try out on those holding the purse strings.

1) It’s a great place to hear about what’s new.  Yes you can read about these things in the press releases afterwards but there’s something cool about being the first to hear about new features and new directions.

2) You’ll get a good sense of where the entire ship is headed.  The keynotes do a good job of laying out a vision bigger than just a product here and a product there.  You can really see this “Enterprise Control System” strategy coming together.

3) The networking is great!  No, not the wireless networks… the people you geek!  Where else can you get the chance to sit down with product managers, company directors, big thinkers, and other interesting people.  What’s cool about this is that you don’t really have to do anything special to get these meetings.  When you get your breakfast or lunch, paruse the room for folks with interesting name tags and sit down next to them at the table.  99 times out of a 100 they are very interested in chatting up the customers or the integrators.  You would be amazed at how receptive they might be to your issue.  I had a laundry list of issues that I presented to someone in the management realm a few years back.  Within a few months that list had turned into a formal internal work list and some of those exact items I complained about are now showing up in future releases (keeping mum on exactly what these features are because I probably shouldn’t know about them anyway )

Aside from networking with Wonderware folks there is also the chance to chat up other customers and integrators.  I’ve made quite a few connections with people at the conference that I stay in touch with to this day.  It’s kinda neat to watch everyone grow through the years as we all get a little older and wiser.

4) The Opryland is an AMAZING place.  Unless you really want to, you literally won’t need to leave the property during your entire stay.

5) It’s not like it’s Vegas or something where you can get in trouble.  All in all Nashville is a pretty serene place, not too many places to tear it up.   On the other hand if you want a fine steak make sure you make your way to the Stockyard.  Just be careful not to park in the empty spot right under the tree up front.  If you didn’t notice the spot covered in white droppings from the birds you’ll certainly notice it on your car when you return.  Also, if you like to pluck the strings, Gruhn Guitar is a must visit.  I’ve been a couple times and even for a rank amateur like myself they were great guys and welcomed me in to play whatever I wanted.  I did get the chance to play a $20K Martin.  Didn’t sound too great as I was a tad nervous you could imagine.

6) Free Training?  Did I say Free?  It’s not quite the full week training you’ll get from your local distributor or in Lake Forest but they have numerous 2 and 4 hour training sessions to get you up and running on a number of different products. 

7) Meet your favorite support engineer.  I’ve talked to so many of them over the years that it’s pretty cool to actually meet some of these folks in person like Chris Azer, Sri, Bindya, Alicia Rantos, Nitin, Edward Xu, and a whole host of others.  They feel like old friends

Finally I haven’t even gotten to the actual presentations.  Some of these are really really good.  I still learn cool tricks every time I go.  There is usually a good mix of Wonderware proper presentations, customers, and integrators.

I hope some of these tidbits will help you over the fence if you haven’t quite decided whether you should go.

- Andy

For those of you who are on the email list for Ops Manage updates you may have seen this little nugget come across the interwebs.

We’re pleased to share just some of the product updates, enhancements and applications being featured at the conference:

Wonderware

· System Platform 2012: newest version to enable an enterprise control system

· Wonderware InTouch 2012: continuing innovation in HMI 

This is exciting, a new major rev for System Platform.  Maybe I’ll poke around a little bit and see if I can dig up any goodies for you ahead of Ops Manage.  While I didn’t participate in a recent InTouch Beta I did watch the bug list and discussions go back and forth.  There appear to be some really neat new features coming down the pipe.  One in particular is something that people have been clamoring for for years now.  I’m not going to disclose this feature as I’m definitely not authorized to speak on it.. but I promise you’ll love it.

- Andy