The ArchestrAnaut

Sometime during the middle of last week one of my customers had a network card fail on them.  No big deal.  We’ve got redundancy using teaming on the network cards so we kept humming along without an issue until we got a chance to work on it, which happened to be the same day.

As an aside it happened to be a terminal server, which in this particular environment is a who cares type of thing.  I say this to force you to think about your own environments.  In this particular facility if an operator can’t see something for 10 or 15 minutes it might be an irritation but it’s not cause for a riot.  However, losing process data is a totally different story.  This data is what substantiates the fact that conditions in the facility were and are under control and within limits.  If we don’t have this data we might not be releasing product… and that’s a bad thing for everyone.  I contrast this with a previous stint in a major chemical plant where if the operators lost visibility we went into full blown meltdown mode, not it wasn’t a Nuclear facility but we panicked.  Depending on the plant you could lose everything in a matter of minutes if an abnormal condition came up and you didn’t deal with it in a timely manner.  There was a particular incident where we lost power to the control room (UPS didn’t work either) but the control cabinets stayed up and kept the process running.  After about 5 minutes the decision was made to drop the plant using the big red button on the emergency shutdown system.  It was much safer to bring the plant down using the ESD than allow some condition to run away while we were blind.  All of this just to say there is no one size fits all answer for where you should have your redundancy.  It should be truly driven by your specific requirements in your facility. Read the rest of this entry »

Depending on your requirements the concept of system redundancy and resiliency might never cross your mind.  For many facilities, however, having a system that minimizes outages and hiccups to an absolute minimum is a must.

When you talk about redundancy you really should look at two different aspects; redundancy and resiliency.  What’s the difference between the two?  For me redundancy is having functions duplicated across multiple components so that if a single component fails the system continues in operation without anyone even noticing.  Resiliency is the ability of a system to easily recover from a failure.

Read the rest of this entry »

Andy & I have always been fairly frustrated with the Wonderware Alarm Logger service.  It almost seems like an afterthought for System Platform (just look at the service name:  New_AlarmLogger).  Perhaps the most frustrating part is trying to integrate it into a system with redundant App Servers.  To the developers:  please integrate this into the System Platform in a future release.

Read the rest of this entry »