Automation upgrades don’t have to be painful.  We’ve all worked on projects that don’t go as planned due to a multiplicity of variables that affect a successful implementation.  To minimize these issues use what you, and others, have learned from mistakes made on retrofit projects.  Too often automation upgrades go something like this:

• Attend Control System Vendor Dog/Pony Shows
• Select a Control System Vendor & purchase system
• Start Programming
• Encounter Problems
• Work around the problems,  soldier on, get it done
• Struggle though startup
• Patch, tweak, and eventually make it work

 To help minimize issues on your process upgrades you must recognize and incorporate the following:

• Know the drivers that determine a retrofit project’s success
• Understand that the control system selection is determined by the drivers and not the other way around
• Know and identify Retrofit Project “Red Flags”
• Execute the project using proven implementation techniques

It is important to recognize that certain project drivers dictate the requirements of the control system.  You cannot select a system until you know what is required of it!  You must know the Stakeholders and their needs. Determine the Project and Schedule “Type.”  Understand what your current system does and what the replacement NEEDS to do. Evaluate your control system options, and choose the one best suited for your situation.  Finally use best practices to execute your new system. 

Over the next few blog posts I will cover tips from the experts at upgrading automation systems.  Most of this information on this series comes from a recent ISA presentation by Hunter Vegas with Avid Solutions. 

Do YOU have pointers, tips or tricks learned from automation upgrades that you would like to share with Automate-ZEN? If so submit them to sbernard@avidsolutionsinc.com and I may use them on these postings.

Zen Habit:  “A wise man learns by the mistakes of others, a fool by his own”“

A Latin Proverb

Alarm clocks are aggravating to me!  Not to mention that annoying SNOOZE button.  What good does that function have anyway?  I never go back to sleep after hitting it and this only causes anxiety with its relentless pursuit of getting me out of bed.   The anticipated benefit from another 10 minutes of blissful slumber is negated by the agitating Buzz-Buzz-Buzz.  Here are two things to remember if sleep and personal time are important to you.  #1 the snooze function only prolongs the inevitable (you still need to get up and start your day) and # 2 life is less stressful without snoozing. 

This phenomenon is also true with your Automation Systems.  You can continue to put off the much needed upgrades (Snooze) but that is sure to keep you up at night and make your life more stressful.  Automation upgrades are daunting and often approached in the same manner as retrieving your ice chest from a sleeping bear.  Reduce your stress and consider going ZEN with a simple and modular approach to your automation upgrades.  The Zen approach will preserve your sanity, your sleep and your relationships.

Join us next week where I will begin a series on Managing Automation Retrofits.  The blog will refer to a technical seminar that I attended at ISA Automation Week in Mobile, AL.

ZEN Habit:

“Zen is not some kind of excitement, but concentration on our usual everyday routine.” – Shunryu Suzuki

Check out this very cool short video from my colleague Jimmy Asher on his Level3 blog  giving an overview to  the parts of  ISA-95 .

Quick Overview:  the Parts of The ISA-95 Standard

National Geographic aired a Christmas day special called Brain Games that was quite intriguing, especially the segment that covered Awareness.  During the program, it became apparent to me that senility was setting in and perhaps it was time for psychoanalysis.  The TV special showed several awareness tests that caused me to understand that I don’t pay attention very well (darn my wife was right again….).  I felt better though as the program revealed, regarding all the data that we process each day, that most humans have a short and selective attention span.

 Researchers at the University of Pennsylvania have calculated that the human eye can transfer data at the rate of approximately 8.75 megabits per second—roughly triple the speed of the average internet connection in the United States. Since humans cannot process everything at once, we tend to prioritize things based upon our surroundings, preconceived notions and our experiences.  The problem is that we miss a lot of what is actually going on around us. According to this program, our sense of sight is the key to our important decision making process.  Roughly 30% of neurons in the brain’s cortex are devoted to vision, compared with 8% for touch, and 2% for hearing.  However, the fact is we miss obvious things in our lives every day and some may cause serious consequences to ourselves, our equipment or the environment.

I experienced a real life example from 1988 through 1990 as I acted as the Division Field Safety Committee Chairman for Chevron.  We decided to have operators visit different facilities than their own and evaluate them for safety hazards.  What a success!  During that period we achieved over 2 million man hours without a loss-time accident. The visiting operators were able to see hazards that we were either not aware of or had become accustomed to due to conditioning (i.e. we stepped over the same trip hazard for over 10 years).  This exercise reduced accidents and saved Chevron a lot of money (I got a clock!).

Zen tip:  Invite someone to your facility (a subject matter expert is best) to visit and evaluate your processes and applications.  With the right person/s they are likely to discover issues you may have overlooked.  This simple exercise could save you money, equipment failure or possibly your life.  You may even get your own clock!

Nooooo.! Not another Three Letter Acronym (TLA).  Don’t we have enough to remember? Let’s consider a TLA that adds REAL VALUE to your business.  How about EVO?   Explore Value Opportunities is soon to be the latest craze and most beneficial TLA for the automation and controls world.  I recently had the pleasure of windshield time with the Master of EVO, Kevin Trantham, our VP of Operations, and one of the best examples of a value adding subject matter expert (SME) in our business.  I asked Kevin to explain the EVO process and how it benefits our clients: 

The first step of EVO is to research the customer’s product, operations, technology and business philosophy.  Schedule a visit with the customer and the stake holders for that facility.  Get the stake holders talking by asking probing questions such as their area of responsibility, the company’s initiatives / goals and what keeps them up at night.

The next step is a plant tour to EVO together with your client.  Then look for problems and convert them to value opportunities. Some opportunities are evident to all, while others are hidden like a snake in the grass.  Our SMEs often see these hidden opportunities and recommend change based upon their experience. 

Meet with the stake holders again and offer your recommendations with expected payback.   The goal at this point is to earn the customer’s trust.  We target areas that will have the greatest impact for improvement with the quickest ROI.

 Here are examples of EVO from existing clients: 

• Increased through put at Dell, Uniboard and Domtar by debottlenecking sections of their operations
• Increased production of the calciners process for ore while reducing emissions at PCS Phosphate
• A major commercial vehicle client realized a $150K/year energy saving by  installing our dynamometer regenerative system.

Once we implement our EVO recommendations, it is easy to monitor how effective they are and begin to realize the payback.  In today’s LEAN environment it is vital to offer EVO solutions such as these.    By Exploring Value Opportunities, your client’s plants run better and you acquire a Trusted Partner.

We have all done it at one time or another. Someone asks, “Is this the latest version of the program?” We say “no but that’s ok, I have a copy on my USB drive”; you plug it in and start the down load, not thinking that you may unintentionally cause a catastrophe!

 I attended two cyber security presentations during Rockwell’s Process Solutions User Group and Automation Fair in Chicago recently where Marty Edwards – Director of Control Systems Security Program (CSSP), National Cyber Security Division U.S. Department of Homeland Security (DHS) and Mark Heard -Systems Associate, Eastman Chemicals really opened my eyes to ICS security risks. The theme of the presentations was Control systems are the only place where cyber attacks become physical. Wow! When you think about it that way it’s terrifying and not very ZENlike.

 We believe that we’re ok because most legacy systems are difficult to hack and today’s systems are assumed to be secure out the box. However, Stuxnet attacked XP and new open systems have wireless, Ethernet, remote access, plug & play etc. and actually increase the risk for attacks. Remote access and Laying Bait are two primary methods to corrupt a secured network. Laying Bait is the easiest and quickest method by creating a backdoor into the network. Remember the credit union attack where USB drives were left in the parking lot? Employees plugged the drives in and within a few hours the “adversary” had entry into the network.

Control systems have many dangerous elements that can cause physical consequence to people, the environment, and equipment. The risk is real! Since 2006 the US Government Accountability Office noted a 400% increase in Cyber attacks (some on ICS). The average cost of an attack is $850K. We need to build a culture of security that includes End Users, Manufacturers, OEMs, Vendors, and Service Providers. ZEN-solution: Check the vulnerability of your control system and what you should do to prevent attacks by visiting the DHS National Cyber Security Division’s CSSP website.

These were the thoughts running through my mind as I attempted to learn my first Kata. The purpose of Kata is to perform the techniques learned in martial arts in a choreographed imaginary exercise to prepare the student for attacks from multiple opponents. I knew the techniques but could not coordinate the moves so I would not embarrass myself. Even worse would be the complete humiliation to my instructor, Sensei, and the entire martial arts world. Sensei pulled me aside and told me that this was my Kata and not to worry about what other students’ Kata looked like. “Just break it down and practice each move until you perfect it.” That actually worked. However, it wasn’t executed as elegantly and technically perfect as my two daughters. Too many automation projects are like learning Kata. That is why so many of these projects overrun budget, schedule and everyone’s patience. Automation projects are often complex, over-engineered monoliths that cause us to lose sleep, family time and eventually our religion. While getting everything you wanted in an automation project is a euphoric goal, often the realization is that it does not offer reasonable payback to the owners or mankind. Apply Sensei’s lesson and reduce your risks by breaking your project down. Evaluate where a small investment with a rapid return will set the stage for a palatable series of modular automation upgrades. Wise engineers and managers instinctively practice Automate-Zen to instilled modular harmony to their lives.

While sitting on the tarmac for a flight from Charlotte to Mobile to attend the ISA Fall Leaders Conference, the young man sitting next to me appeared to be quite nervous.  Sweat was dripping from his brow onto his shaking Sky Mall magazine and his knees were moving like a full steam locomotive.  I asked if he was ok. He told me that he was frightened and was afraid to die in a plane crash.  I explained to him that flying is safer than driving a car.  Looking for a way to comfort him I did say, “Besides, you’ll only go when it is your time.”  He looked relieved for about 2 seconds then started shaking again and said, “What if it’s your time?”  Ok, this is actually a joke. However, on that flight I did inquisitively watch the pilot and copilot carefully go through the preflight safety checks (why was the door to the cockpit open?). They were moving switches, pushing buttons, etc. and checking off the list.  This exercise sparked a memory from a presentation I attended just a few weeks earlier on Situation Awareness. The title slide read “Our Operators Are Our Major Assets” and the next slide read “they are also our biggest liability”.   Wait a minute, the pilots are the operators and …………….ugh.

In this presentation to our local ISA section, David Lee from User Center Design Services  explained that most equipment we use for process safety systems are very accurate. Then he goes on to say that, it is human error which causes or contributes to most shutdowns, loss of production and accidents.   The following is an excerpt from his presentation:

So is 99.99% error-free good enough?  Below are real world applications of what ONLY 99.99% error free could be:

• 2 unsafe plane landings per day at O’Hare Airport
• 500 incorrect surgical operations each week
• 50 newborn babies dropped at birth by doctors everyday
• 22,000 checks deducted from the wrong bank account each hour
• 32,000 missed heartbeats per person, per year
• 114,500 mismatched pairs of shoes shipped each year
• 200,000 documents lost by the IRS this year (I think they lose more than this!!)

Want to guess what the average cost due to human error in US & UK industries are? $30 billion per year!!  Three Mile Island; Exxon Valdese;  Texaco, Milford Haven; Esso, Longford, Australia; BP, Texas City; Northeast blackout;  Chernobyl, USSR; Buncefield, UK, all contributed to by human error.

 Sure there are a lot of Standards and Guidelines available to mitigate risk of Human Error: ISO 11064 (CR design, HMI); ISA 18.2 (Alarms); ISA 101 (future) (HMI);EEMUA 191 (Alarms);EEMUA 201 (HMI); API 1165 (HMI);API 1168 (Alarms) and API RP 755 (Fatigue) to name a few. 

 But the root of Situation Awareness and risk mitigation is ZEN.   A simple 4 step process:

1. Detection
2. Orientation
3. Diagnoses
4. Action

 Situation Awareness tips:

• Look at the BIG PICTURE
• Foster a culture that prioritizes situation awareness
• Keep things simple
• Reduce stress

 Simple huh? Not so fast, I do recommend hiring a professional to evaluate your risks and implement situation awareness no matter how well your system works.

Here is part 2 of 2 of my interview with P. Hunter Vegas regarding the benefits of staging automation projects

1. So after drawing the process, reviewing as a team and resolving any problems that arise do you still use the whiteboard?
1. Yes, we leave the whiteboards and process sketch up the entire project.  As we go through the project the team will check off the areas of the process that are complete.  This exercise gives us another sense of accomplishment.  Small successes such as these build confidence and knowledge of what is complete and what remains to be done.

2. Are there any downsides to staging a project? For instance personal calls, emails and the team exposed in this open area, etc.?
1. This model is usually more efficient but there are a few downsides and negatives that may not make it (staging) appropriate for everyone.  There are drawbacks that must be managed.  For instance, if you are working on very complex code and need to absolutely focus on your task, having 3-4 people in your space constantly asking you questions is sometimes disruptive. You have to remove yourself from the area to accomplish these tasks.  There are also personality differences and some outside interruptions from others not working on the project, but this model is absolutely best for us.  These distractions are significantly offset by the added efficiency the team has executing the project.   We all have offices but we spend 95% of our time in the staging area. 

3. What projects or situations would not lend itself well to this model?
1.  It (staging) works best with a large project with a large project team.  This is true especially with a diverse mix of experience from the team members.  If you have very seasoned engineers who have tasks that can be worked on independently then brought together at some point you don’t need to stage the project team.  However, most times we have both seasoned and younger engineers working on the project doing the same things,  so staging works best for us and probably most others as well.

Right outside my office door there is a large project staging area, usually staged with workstations of a DCS System for one of our clients.  The area is typically occupied by Hunter Vegas, one of our Project Leaders.  Hunter has his team work full time in the staging area for good reason, it is very efficient!   I interviewed Hunter and will post two separate blog posts highlighting the value of Staging Automation Projects.

Interviews:

Interview with P. Hunter Vegas regarding the value of “Staging Automation Project Teams”

1. Is there any correlation to successful projects and the staging of the team in one area.
   1.  There is certainly correlation when we stage our project team.  Most programming is complicated and involved.  Even though things are laid out well they are sometimes not clear.  Bad hand writing, questions not understood and cryptic notes are better explained in a group setting.  Having someone 5 feet away as opposed to them being in another room allows them to immediately ask questions for clarification.  This is quite helpful but the thing that’s more important is that everyone hears the answer.  If they happen to run into the same issue the resolution is known.  The setting also lends itself to active and productive conversations and really  minimizes wasted time or confusion.
2. I noticed you have a number of tools in the staging area that are utilized during the project such as white boards, workstations, etc.  How are they incorporated into the project.
   1. We use large white boards in the staging area and the 1^st thing I do is sketch out the entire process and take an hour or more to go over the process and our project scope with the team.  This gives everyone the understanding of the process and what we are programming.  Any questions are answered here and while reviewing the process if any errors or confusion arise the team works through it all before beginning to code.  Another aspect of using the whiteboards is that when mistakes are found during the project they are put on the board and when corrected and tested they are checked off.  This becomes a running check list of what needs to be done and reviewed.  This lets everyone see issues in writing and usually eliminates making the same mistakes again.  Writing the problems down and checking them off when resolved gives everyone a sense of accomplishment and makes the first time right much higher.