We have all had that wonderful experience of getting to the end of a project, and stepping back to admire the glory after months of hard work. After all the hours of programming, testing the programming, getting new information from the customer so that it’s necessary to go back and change the programming – the end is in sight. At this point, for most projects, there still remains the Factory Acceptance Test (FAT), where we get to show off all that hard work in front of the customer. During this time, we must summon the fortitude to remain sharp to carry the project over the finish line mistake free. With this in mind, I would like to share a cautionary tale of an FAT that ended much like the Olympic hurdler face-planting on the final hurdle.
For most projects, pre-FAT testing involves simulation that is virtually accomplished through software and not actual hardware. This is all well and good for logic testing, and even HMI testing virtually is fundamentally safe, but when it comes to actual hardware testing, the inadequacies are obvious. Therefore, while our FAT design is largely derived from our pre-FAT simulation testing, we are left to conceive of a functional design to test the hardware. It is in this design phase where we once again must put on our engineering hard hats when it is so tempting to think they are no longer needed.
In this particular situation, I was approaching the hardware FAT much like the software FAT. My thought bubble was saying, “All I need are some screens that quickly show an Input turning on a corresponding Output.” The customer could easily see from the HMI, for each digital output channel, the little circle turn from red to green when the channel was turned on. I was even prepared to point out the channel LEDs on the output module to show the channels turning on and off. At the time, for a simple hardware check, this seemed adequate.
If this were a movie, we are at the point where there would be a dramatic flash forward scene. Picture this, I am standing with the customer and other contractors and we are ready to start the system up. This is a waste-water expansion project, so we are filling up the sump tank to reach the high level trip point, and when we do, low and behold, not one sump pump starts as per the design, but both the primary and backup pumps start. Since I have extensively tested the logic to know that only one pump should be running, while also confirming that only one pump is being commanded to start - this led me to conclude that the electricians must have wired something wrong.
Let’s fast forward through the frantic troubleshooting efforts that took place next of draining the large tank, tracing and checking the pump wiring, triple checking the logic, and visually seeing only the one channel LED illuminating on the output module. At that point, I was yet to arrive at the culprit. But, it was then meter time where every system integrator gets to impress the electrical contractors, or plant electricians, and get some “street cred” for not just being a programming nerd.
Again, if this were a movie, the dramatic music would slowly build as I traced with the meter from the module down to the terminal block, only to reveal the flaw in my hardware FAT design. Although the logic was not turning on the backup pump output channel, and although the output module channel LED was not on, and although my HMI screen only showed one channel at a time turning on during the hardware FAT, the meter proved that both channels were on from the terminal block. So, the culprit was none other than a faulty IFM cable that had a short between the module and the terminal block assembly.
You may be thinking at this point, “Yes, but this was beyond the controller hardware because it was an IFM cable that plugs into the module and extends to the terminal block.” I confess that that was my knee jerk self-defense statement at the time, but as I thought further about the hardware testing design, I began to see how relevant this was to even the most basic hardware architecture.
First of all, the hardware for this project was all contained inside a control panel, and it should have been tested all the way down to where the outside connection was made. Secondly, even if the hardware scope only extended to the local chassis, channel shorts could still be missed if a simple hardware testing design were used that only looked at one channel at a time. After all, before finding the shorted IFM cable, my first thought after seeing both channels on at the terminal block was that possibly a metal filing had gotten between the channels because the electricians had to drill another bottom hole in the panel when it was brought on site.
To bring this cautionary tale to a close, I will summarize what I learned from this. First of all, I learned to not think of the hardware side in the same way as I do the software side when it comes to testing. Thinking this way led me to oversimplify the hardware testing and rely on testing methods that are better suited to the software side. Secondly, I was reminded that the signal should be tested at the scope end point. I needed to determine at what point the signal left my scope and became someone else’s responsibility. This is common sense, but can be forgotten when the temptation of quicker and easier, yet less reliable methods are available. Third, I needed to watch out for the dreaded short. Testing only one channel at a time, even with a meter, can lead to missing shorted channels. For digital hardware, taking the time to devise a test for all channels to ensure that no shorts exist may add some time, but could eliminate a costly error.
My final and most important learning is that after all the endurance it takes to get through the initial programming and pre-testing phase of a project, the project is not truly over until the system is installed and performing to the customer’s expectations. The same level of diligence is needed to carry the project over the finish line each and every time.