Getting the project right

Greg: A successful automation project depends more than most realize upon capturing the expertise and knowledge of many people and keeping them involved. While we recognize the need to include the input of process engineers and operators, we may not realize that we need to make the most of the knowledge of those responsible for safety, control strategies, Digital Twin modeling, configuration, operator interface, alarm management, Instrument and Electrical (I&E) design, and maintenance. Increasingly, these people tend to be at system suppliers and engineering offices as well as the plant. To get all of these people with diverse skillsets and at different locations literally and figuratively on the same page requires documents capturing the intent and details of the automation system that are understandable and concise, yet comprehensive. Documents tend to reflect the knowledge base of the originator, not taking into account the wider perspective needed for effective communication.

Mike LaRocca, a process control specialist at a major specialty chemicals company who I worked with during my days at Monsanto and Solutia, has effectively addressed this issue. Mike has developed a key series of documents for an automation project.

Mike: In the Mike LaRocca world of control definition, there are five key documents, and they should, in general, be developed in the proper order (though in reality, it is kind of an iterative process because sometimes you figure out things need to work differently as you get into the details). In my book, the combination makes up a clear and complete Functional Requirements Specification (FRS). Having all these documents provides a way for us to think through the details and make sure we get exactly what we want. I think organizations should strive to develop standards defining these documents, but getting everyone to agree on a format, content, symbology, etc. that fits different situations and different control system types is, admittedly, pretty difficult.

Greg: What is the first document?

Mike: I recommend starting with a high-level design document called a Process Logic Description (PLD). This is a process engineer’s description of how things should work (i.e., “open valve A, then start pump 1, wait until tank level reaches X…control tank temperature by controlling the flow of glycol to the jacket…cascade the tank temperature controller to the jacket temperature controller)—simple process-oriented descriptions. They don’t necessarily even need to reference tags because tags may not have even been assigned yet. PLDs should not include much control jargon and terminology, and they should be control-system-independent (i.e., should apply whether you are using a PLC, DCS or some other control system). PLDs are the first pass at what is wanted, at least in the eyes of its authors. It then can serve as a guiding document and get the juices flowing for the people who develop the more detailed documents.

Greg: How do you address requirements to deal with abnormal operation and ensure safe operation?

Mike: While dealing with abnormal situation management and safety is a multifaceted exercise, and portions of it end up in the process design, much of the rest of the output from that exercise ends up in the second document, the Interlock List. I’m pretty picky about interlocks—how they are designed, documented, tested, etc.—because in my experience, poorly designed interlocks are the things that cause the most heartburn during a startup and sometimes throughout the life of a system. If you aren’t careful, you end up boxing yourself in a manner that makes other aspects of control difficult to implement well. It’s really important to think about what you are protecting against with an interlock, and figure out the least disruptive way to implement it while simultaneously keeping it simple, easy to check, easy to document and easy to maintain.

It’s also important to be very specific about what instruments and field devices are involved, and how it’s going to work. What measurement is going to serve as the tripping device? Specifically, what instrument—maybe you have two level instruments; a transmitter and a field switch, for instance. What is the trip point? For a multiproduct facility, does that trip point need to change with the product or grade? When the interlock clears, should the process or piece of equipment automatically restart or reopen, or does an operator need to check things out and restart them? What dead band should be applied for clearing the interlock so the interlock doesn’t chatter?

In one of my former jobs many years ago, we had a nice thick document called a Standard Manufacturing Process Description (SMPD) for each of our processes that was basically supposed to be the bible for how the process operated (or should be operated). This was in the days before DCSs and even PLCs to some degree—almost all the interlocks were hardwired. The SMPD did a great job of describing the chemistry and the major unit operations, however, it gave short shrift to interlocks and the table provided listed interlocks very generically: “The high temperature interlock closes the steam valves.” It didn’t identify what temperature instrument served as the trigger (we had multiple temperature measurements), or what steam valves were actually closed. It wasn’t really even clear on what the trip point was supposed to be, because that information was buried in verbiage elsewhere in the document, and different values were mentioned in different sections of the document. When it came time to retrofit the process with a DCS, I had to dig into all the electrical schematics to figure out how things worked, and had to test the switches in the field to figure out what the trip points actually were. It turned out there were three different high-temperature interlocks, but you wouldn’t have known it from the lame interlock descriptions in the SMPD.

On retrofit projects, it surprisingly is not all that unusual where I had to ask why an interlock even exists. Sometimes, even the long-time process experts are a little confused. Is it a safety interlock? Property protection? With a little probing, we usually figure it out, but why not be clear about it upfront and eliminate the uncertainty?

I now ask for the following features in an Interlock List:

1. Each interlock needs a tag, or unique identifier—a handle, or shortcut, if you will. This gives you something to refer to it by in other documents, or maybe a preventive maintenance (PM) work order, or even just in conversation. It’s a lot easier to convey exactly what interlock someone is talking about when they say, “We need to test interlock 101-01 and document the results” than it is to say, “We need to test the high-level interlock on reactor 101.” Why? Well, because there could be two high-level interlocks on the tank, one triggered by a transmitter and one triggered by a level switch. Or, maybe one is hardwired, and one is implemented in software. Having a clear, unique identifier eliminates all ambiguity in communications and documentation, just like having a tag on an instrument does. It’s like what a doctor I know did for identifying all his examination rooms: He had the door frames all painted a different color, so after you check in, they say, “The doctor will see you in the red examination room.” They don’t have to say, “Make a right at the end of the hall, then pass the drinking fountain and go into the room on the left.” You just walk down the hall until you see the red door. It’s simple and there’s no ambiguity.
   
   I like to tag interlocks with a number that in some way associates it with the major piece of equipment it’s designed to protect. So, if we are talking about a high-temperature interlock on reactor 101, I would give it a tag something like I-101-01. And if we have redundant, but similar, interlocks, I might tag one as I-101-01A and the other I-101-01B. Then, the next interlock on that piece of equipment might be associated with a different process measurement, say, level, and it would be I-101-02. Using suffixes like A and B help keep similar interlocks close to each other when sorting the list by tag just for organization’s sake. It’s also not a bad idea for some processes, for organization's sake, to reserve suffix numbers for different process measurements. Interlocks with suffixes -01 through -03 are for level (maybe you have two low-level interlocks and one high-level interlock), suffixes -04 through -06 are for temperature, and suffixes -07 through –08 are for pressure. If you adopt such a convention, then over time people pick up on it and immediately know that when you mention interlock I-101-05 or I-102-05 that you’re talking about a temperature interlock. Going to this extreme is not necessary, but it is a means of providing structure and consistency, and those are two great aids when it comes to communicating and training people.
2. A clear purpose description detailing what event are we trying to avoid and what are we protecting. This helps when it comes time to design interlock implementation including what level of reliability is needed. If an interlock has a safety purpose, it very well may need to meet different requirements than an interlock whose purpose is equipment protection. A clear purpose also helps years later when someone wonders why the interlock exists to begin with, and why it was designed the way it was. Sometimes the purpose of an interlock is not always clear based on just a description of its trigger and action. Notice, I said the event we are protecting against also needs to be identified. That may not be evident just from a simple purpose description. Saying, “Prevent injury from an explosion” doesn’t necessarily identify the event I need the interlock to interrupt. There could be multiple events that could lead to a reactor exploding: high pressure or high temperature, for instance. So, you might need to expand the purpose description by saying, “Protect personnel from injury by preventing the reactor from reaching a temperature that could trigger a runaway reaction.” In my book, the more specific the description the better.
3. A clear English-language description (all words; no tags, no symbols) of what the interlock does. What’s the trigger and what’s the action? Preferably, it should be in the form, “If (trigger description), then (action description)” or, “(action description) if “(trigger condition).” For example, “If the reactor temperature exceeds 100 °C, then the main steam block valve is closed and the main cooling valve is opened.” This helps clarify what an interlock is doing, and sometimes clarifies how it should be or is designed. I think it’s important to keep with this kind of phrasing because interlocks are supposed to be simple, direct, and hard lines in the sand, so to speak. If you’re finding this phrasing getting too complex or sounding wishy-washy, then you probably need to rethink the interlock. Maybe it needs to be broken down into multiple interlocks.
4. A symbolic description that identifies the specific control system elements, instruments, signals and final control elements involved. This allows for further specificity in how the interlock works and what is needed. It identifies exactly what instrument and signal (e.g., density or mass flow for Coriolis meters, or middle signal for pH transmitters) is being used for the trigger, exactly what the limit is, and whether the limit is hardcoded or is a variable that can adjusted or made part of a recipe. It also identifies exactly what field device is going to be acted upon by the interlock. Oftentimes, we need fully redundant (that is, separate and independent) interlocks, where we need two trigger devices and two actuating devices. That is, interlock 101-01A is triggered by temperature indicator A, which closes valve B, and interlock 101-01B, which is triggered by temperature indicator B and closes valve B. This makes it clear how to program the interlock (if done in software), or how to design it electrically or pneumatically if it is done without software.
5. Avoidance of “ORs.” If you find yourself wanting one, the issue can usually be addressed by creating two separate interlocks. The reason is two-fold. First, having “ORs” makes organizing interlock PMs and writing interlock test descriptions and recording results more difficult and complicated. In a typical interlock test, you set things up—for instance open all the valves that the interlock is supposed to close—then trip the interlock condition and verify all the valves close as they are supposed to. If you have an interlock trigger that has two conditions ORed together, then you basically have to explain that the test needs to be split up into two separate tests, each testing one-half of the run, and test twice—trigger the first half of the ORed condition, then set things up again and trigger the second half. If everything works both times, or if it fails when either condition is triggered, no biggie—you record “Passed” or “Failed” and move on. If the interlock fails when triggering one-half of the ORed condition but not the other half, then recording the results becomes more cumbersome because you have to explain which half failed. In the era of computerized maintenance systems that are expecting a simple Pass/Fail entry, this can be a bigger headache than you might imagine. It’s much easier to just break the ORed interlock into two separate interlocks.
   
   A second reason for avoiding ORs is it makes it more difficult to indicate to the operator exactly which of the ORed conditions is the culprit. This later reason may have more to do with how I am used to presenting interlock indications on our systems than anything else. Perhaps you could come up with a different interlock indication paradigm that would be amenable to triggers with “ORs” in them, but it seems to me not having ORs in the conditions for a single interlock makes it easier for everyone. Avoiding “ORs” also avoids ambiguity in communications and reporting.

Greg: What’s next?

Mike: Control Module Function Descriptions (CMFDs) and Sequence Function Descriptions (SFDs), the third and fourth documents, come next and in that order. CMFDs are detailed design documents that describe how an indicator, discrete control device or control loop should function. The documents include basic configuration details such as scale, engineering units, fail position, alarms, I/O channel definitions, permitted modes, parameters to be historized, etc. This configuration-oriented information can often be presented in table form in the documents. It’s fairly standard, boiler-plate kind of stuff.

The difficult, and sometimes most important, part of a CMFD is the stuff that goes in the free-form text Function Description section. This section is used to describe programming details and how the module or physical device it interacts with fits into to the big picture. The programming detail might include information about interlock functionality, how information from the module is presented on a display, how it interacts with other modules, or the description of how the module should respond for abnormal situation management, for instance, a transmitter failure. Finally, the free-form text section can also be used to give details on the physical field device that the module interacts with: what does it do, what are its purposes, maintenance problems, troubleshooting tips, possible safety concerns, etc.—anything that would be helpful to the software guy, operator, maintenance tech or process engineer down the road. CMFDs are intended to be living documents beyond just building the system, just as an electrical schematic has a use that goes beyond the construction phase of a project. It’s used for troubleshooting, marking up as part of a change control procedure, analyzing, explaining and training. There are lots of good uses for a well-written CMFD. Since it has an ongoing use, I always write the free-form text section in the present tense. You won’t see “shalls” or “musts” in the CMFDs I write, because I want them to be thought of as more than specification sheets to be used once and then disposed of. An example of a CMFD is 0260-AIC-101-08. 

Incidentally, lest I forget, also like an electrical drawing, we include a revision table in our CMFDs, complete with revision numbers and a description of the change made. This works well with our management-of-change process.

The control modules in a control system are the core building blocks. They are the glue that holds everything else together, and as such, I probably spend more time developing CMFDs than any of the other key documents. Many other parts of a control system are going to rely on them, or be built on top of them (interlocks, equipment modules, phase logic, display elements), so it’s important that they’re thorough and you get them right. I do a lot of thinking and questioning of myself and others when writing CMFDs to be sure I’ve covered everything, and that I’ve provided ample features and flexibility. I want them to be rock solid and hiccup-free. I don’t want the software coder reaching the point of writing phase code only to find out that some important functionality was left out of a control module because I didn’t cover it in the CMFD. I try to think ahead and include things that people might want in the future, but which may not be key to the project at hand. For instance, might we someday want an hour meter in the module for that agitator motor even though we currently don’t have a predictive maintenance program? Adding one just might be the thing to the reliability engineer was looking for, but never thought to ask for. Or, I might be unsure how well a loop will handle a step change in its setpoint, so maybe I’ll include a setpoint ramp function in the control module just in case I have to resort to using it. In my experience, it’s usually a lot easier to add functionality when the coder has a blank slate than after he has started coding. Just a caution with adding extras—you have to be careful of scope creep, so it’s best to limit the extras to small, easily coded things.

Greg: And what about SFDs?

Mike: SFDs are detailed design documents that cover batch, phase, step and sequence logic. They describe the sequential actions for running an automated batch-oriented process: “Open valve A, start pump B, wait for the level to reach X, etc.” For multi-product/multi-recipe batch processes, the SFDs also need to identify all the recipe parameters. For a greenfield process, SFDs are pretty tough to get right the first time because often, you’re just making an educated guess at how things should or will work. There’s a lot that you may not know at that point. You may not know, for instance, how long it’s going take to fill, heat or mix a particular tank until you actually start running the equipment. As a result, you need to be conscious of what you don’t know for certain and build in flexibility. In my experience, rigor pays off at this stage of the control definition process, so taking a deep dive into to all the details is usually worth it. Multiple reviews with multiple people at different times is the best way to avoid those forehead-slapping moments during startup where you say, “Well that was pretty dumb! How did we miss that?” and then find yourself having to rewrite a bunch of sequence code. If you don’t find yourself modifying SFDs multiple times during the control definition phase, then you either have a very simple batch process, or you’re not thinking things through enough.

The difference between a PLD and SFD is that the SFD covers all the detail needed to program the action/task, and is control system specific (takes into account the underlying control system nomenclature, terminology, rules, etc.).  For example, where the PLD may say ““transfer the contents of T-101 to T-102”, the SFD would say:

Set Mode:XV-101-01 to CAS

Set Mode:P-100 to CAS

Set SP:XV-101-01 to OPEN

Wait for PV:XV-101-01 = OPEN

Set SP:P-100 to START

Keep in mind that figuring out what phase or sequence logic should do when everything is running wonderfully is less than half the battle when writing SFDs. You also have to address abnormal situations with the logic you develop. Sometimes, dealing with abnormal situations can be handled with simple branches in the main logic. At other times, you basically have to take the process to a safe hold state and let the operator sort things out manually, then restart the batch. It’s that part of the effort that can really tax one’s brain and imagination. You have to not only think about what actions you need to take in your hold steps, but you also have to think about how to smoothly re-enter the normal execution path. Thinking about this early on is more apt to produce sequence code that flows smoothly and is easy to follow, troubleshoot and modify than waiting until code writing has already started. You should strive to identify the re-entry points early. You should also think about how to facilitate manual re-entry (often referred to as an active step change). Sometimes, when running a batch or sequence, you may need to manually step backward and re-do certain steps because of problems encountered. The more you can do to facilitate active step changes, the less likely someone is going to screw up when one has to be performed. A lot of this amounts to managing unit variables properly so you are assured when you step backward that some variable that specifies an important control variable gets set to the right value. The setpoint for a tank temperature controller is likely to be different during a cooling step than in a heating step, so if the normal order is to heat and then cool, but something happens such that you have manually step back to the heat step after reaching the cool step, then, ideally, you would like ensure the manual re-entry point includes the logic for getting the right temperature setpoint loaded into the control loop. It takes a lot of careful planning with regard to what tasks perform what functions to make sure the active step change doesn’t go badly.

Greg: What about operating display graphics? How do you make sure you get what you want there?

Mike: In the old days, I used to sketch up exactly what I wanted the HMI graphics to look like and included notes about how various attributes were to be expressed on the various graphical elements. For example, when should the color of a pipeline shown on the graphic change color? These days I almost never do display sketches anymore, and instead provide the fifth document, a Display Description. We have enough examples on existing systems, and also have built up a good a library of display elements that are used on just about every project, so I no longer need to give a lot detail on that front. I suspect that’s true for many other end users as well. I make up a list of displays with a short description of the content that should be on them, then I point the software guys to the project P&IDs for guidance on how everything fits together. Sometimes, I also mark up the P&IDs to indicate what elements, or what sections of a P&ID, should appear on specific displays. I supplement that with comments in the CMFDs, Interlock List and SFDs about how I want things to look and function. The key thing in display design is to think about what makes for a good operator experience, and avoid arbitrary replication of the P&IDs. The odds are that whomever developed the P&ID was not giving a whole lot of thought to control system displays at the time, so it’s the control systems engineer’s job to compensate for that. It’s important to organize things so the operator doesn’t have to constantly bounce from display to display to run the process or to figure out what’s going on.

Display specification and documentation is one piece of the control definition document set that I don’t view as living document, so I typically trash any lists or sketches at the end of a project. The display itself serves as the living document. It’s pretty easy to print out a display and mark it up when it comes time to make modifications or include it in a training manual.

Greg: Are there any parting words of wisdom?

Mike: In my experience, control system design is a pay-me-now or pay-me-later proposition. It’s a lot less time-consuming, less expensive and less stressful to fix problems in the design phase than it is during startup. Getting everything down on paper where it can be reviewed thoroughly before any configuration or coding starts is a big step toward a smooth startup. Having a methodology and document templates for doing that is key. Also, if you turn the control definition task over to outside resources, be sure you give sufficient guidance up front, plus sufficient time and priority for your internal experts to review everything. Having a standard FRS document set can make the internal review process more efficient.

Greg: To learn how to achieve the best alarm system and operator graphics, see the Control Talk columns with Nick Sands, “Alarm management is more than just rationalization;” with Darwin Logerot, “Dynamic world of alarms;” and with Bonnie Ramey, “Advice on implementing ANSI/ISA standards for operator interface.” For guidance on project management and execution, see the Control Talk columns with Hunter Vegas, “Successful Retrofit and Automation Projects,” “Retrofit Projects - Getting the Flows Right,” and “The Final Word on Instrument Upgrade Projects,” and the column with Denis Backer, “Things to do and not to do in project execution.”

For a more comprehensive look, check out Chapter 10, “Improving Operator Performance,” written by Nick, Boney and Darwin Logerot with additional contributions by Mark Nixon and me, and Chapter 12, “Automation Project Management,” written by Hunter Vegas and Denis Backer in the 98%-new 2019 Process/Industrial Instruments and Controls Handbook Sixth Edition.

For advice specific to safety instrumented systems (SIS), see “The ins and outs of safety systems,” parts 1 and 2. Part 1 explains reliability, redundancy, integrity levels, and testing. Part 2 delves into risk analysis, operation and maintenance.