Mission possible: Control and IT integration

Many large-scale LANs use the network communication access structure shown on the left. On the right, the communication access layers have been reconfigured to retain maximum network uptime and minimized unnecessary network traffic, while adding the real-time response requirements required by instrumentation, control and automation equipment.

Talking and Listening…Really
Sometimes IT issues aren’t so much technical as they’re cultural. Many books explain the value a company can reap by encouraging cooperation across domains. Though some companies are slow to subscribe to such beliefs, Dick Hill, vice president and general manager of ARC Advisory Group, Dedham, Mass., reports more companies are, at least in the area of IT, successfully changing their culture and integrating domains.

“We’re encouraged to see an increasing number of enlightened companies recognizing the benefits of integrating plant IT and enterprise IT into the same department under a common boss,” says Hill. “These companies realize that when people are in the same department and have the same boss, they talk more frequently, they’re more willing to listen to a peer’s explanation of needs, and they’re more likely to work together to develop an appropriate and workable outcome.”

Johan Nye, control systems chief engineer at ExxonMobil, Irvine, Tex., also explains the benefits of plant-enterprise integration. “ExxonMobil is transitioning from proprietary to open digital control, and we’re learning to manage security risks as an integral part of our process control practices. In addition to the technical factors, we’re addressing policy and people aspects, including integrating security into our safety and reliability policies. For instance, the first security priority of enterprise IT systems is usually the data, which often is best protected by shutting down access, but that’s certainly not an appropriate solution for a process control system. It’s fundamental that everyone understands that we’re talking about a process control system that uses IT—not an IT system that does process control. It’s not about protecting the data. It’s about protecting the process.”

Manternach cautions, “There’s a risk associated with reassigning the automation engineer who’s been functioning as the plant IT person to report to an IT boss. Whether it’s stated or not, the message is that IT is more important than improving and optimizing process control. There’s frequently an assumption that the process control/automation engineer wants to be involved in plant IT, when the reality is he or she probably assumed those duties out of necessity. Savvy organizations make informed reassignments of people that produce a win-win-win. Automation engineers are happy to return to doing what they do best; unit managers and operators are happy because the process has fewer excursions, better quality, and greater throughput; and IT folks are happy because one of their own understands the idiosyncrasies of plant IT.”

Jim Sprague, who works for a major middle-east oil and gas producer/refiner, says, “We get great support from the IT folks located at our plants. It’s in the corporate office that conflicts between I&C [instrumentation and control] and IT often occur. For example, we wanted to install a few wireless transmitters, but corporate IT said they have jurisdiction over ‘frequency sensitive equipment;’ thus our ability to explore emerging technologies is sometimes slowed by jurisdictional issues. We’ve just begun to implement a solution we hope will prove beneficial: reassigning a few IT folks to our I&C group with an I&C boss.”

Cross-Training and Hiring
Regardless of how the integration of IT domains occurs, simply reassigning people and redrawing an organizational chart isn’t sufficient to ensure success. The use of open information technology platforms in process control applications also requires rethinking training and hiring practices.

More companies are training process control professionals in IT topics, and educating IT professionals in the priorities and unique needs of the process control environment.

Jack Koziol, of Chicago-based InfoSec Institute says, “We’re seeing an increase in IT professionals attending our SCADA security courses. Many of our students arrive with a wealth of enterprise IT knowledge, but quickly acknowledge that they need to modify policies, thinking, and habits to successfully protect their control and data acquisition systems.”

Cross-training existing employees is one thing, but who you hire in the first place is also worthy of investigation. Nye says, “ExxonMobil prefers that its new hires have earned dual degrees, one in chemical engineering and a second in computer science.”

Invensys vice president, Peter Martin, sums it up this way, “We no longer tolerate islands of automation, and we shouldn’t tolerate islands of culture.”

Narrowing the Plant-Enterprise Gap
Ian Verhappen, MTL’s fieldbus technology director and founder of ICE-Pros, Fort McMurray, Alberta, says, “The gap between plant IT and enterprise IT is getting smaller, thanks in part to enterprise IT folks having to provide high-reliability systems to meet the growing demands of a 24/7 global business environment. From these new business demands, enterprise IT has developed a much keener awareness of the impact of such things as shutting systems down to install a patch or upgrade.”

That’s good news for most companies because, despite the breadth and width of process industries, when it comes to plant IT, most companies face many of the same issues, so sharing best practices benefits everyone.

Plan ahead. Though it’s not always recognized, process control and safety systems get on a project’s critical path at some point, usually just before startup, which is about the same time data historians, engineering workstations and other open platforms are being connected to the corporate network. Informed, integrated IT cultures have already discussed and worked through the various issues necessary to make the connections and maintain the project schedule. The same may not hold true for companies clinging to IT islands.

Mitigate security risk. An effective means of mitigating security risk is to insist on separating the process control environment from safety and business systems. ExxonMobil, for example, insists that firewalls be used to separate basic controls from supervisory control systems, which are separated from the MES-level systems by another firewall. Safety and control system integration is limited to read-only access of the safety data by the control system. “By separating control and safety we may well prevent a security event from becoming a safety event,” explains Nye.

Know the policies. You can save yourself a lot of heartache by learning about, helping to develop, and/or applying your company’s IT policies. For example, ExxonMobil’s policies declare “no Windows” at the basic process control layer, as well as rigorous procedures for managing and controlling anti-virus protection, home-grown node health monitoring agents, and patch management practices for all Windows-based nodes. External network interfaces, including any wireless access points, are centrally managed across the corporation, each having been subjected to a formal risk assessment process. Authentication and encryption are standard features of any external ExxonMobil link. 

Apply common sense. John Rezabek, controls specialist with BP Amoco Chemicals, Lima, Ohio, says, “The biggest issue I’ve had with enterprise IT folks is their zeal for what I perceive as overly paranoid hyper-security. I concede that most IT security is a more-or-less severe speed bump for a sufficiently motivated hacker. The greater threat is likely to come from the disgruntled insider whose login is already past much of the security.” Before insisting certain policies are applied to process control systems, plant and enterprise IT folks should conduct a risk assessment to ensure a thorough understanding of where threats are likely to originate and how IT policies might impact process control system availability and reliability.

Use technology to enforce policies.Policies alone can’t ensure communication networks remain stable and authorized. Need proof? In the 1990s, a number of DCS engineering workstations were connected to “unauthorized” modems. The argument was that these modems allowed the automation engineer to login from home and help the operator in the middle of the night. Eventually many of those automation engineers were “right-sized” out the door, or they simply moved on. Some of those DCSs are still in use, and those unauthorized modems remain connected.

Today corporate IT policies may include a list of approved brands and models for switches, servers, etc. Sophisticated network configuration management technologies exist to monitor network health as well as help prevent unauthorized connections. IT departments use “sniffer” software that knows what brands and models are permitted, checks their configuration, etc. If an unauthorized brand or model is installed the, sniffer software will detect and alert the network police—often within minutes of the device being connected. Likewise, if someone configures an approved brand or model incorrectly, or an unauthorized change is made to the configuration, the network police will be calling.

Embrace ISA S95 IT departments have been integrating business applications, such as ERP, using software models and templates for more than a decade. ISA 95 can provide a similar structure for integrating process and enterprise systems. Integrating process with enterprise systems will be easier if familiar terminology, models, and templates are used. Users who look at ISA S95 “standards” for the first time are surprised to find that they aren’t specifications for defining messages and/or programming interfaces. Instead they learn that S95 is a specification that defines alignment processes. It encourages control-system and related product manufacturers to incorporate S95’s terminology and use one or more of its many models as part of their products’ integration architectures. Even though S95 doesn’t define a formalized protocol for information exchange, organizations such as WBF, formerly World Batch Forum, have created royalty-free implementations of the S95 specification using XML schema definition language. The result is integration models that enterprise IT folks are likely to recognize and more quickly embrace.

Use familiar designs. Designing Large-Scale LANS by Kevin Dooley describes a standard network structure that has been widely adopted by many IT departments. In a paper called “The IT Implications of ISA S95 and ISA 99,” Brandl expands the three-layer network architecture described by Dooley to work in conjunction with the ISA S95 specification. Brandl takes an IT architecture that’s familiar and widely accepted by IT engineers, and explains how to extend the architecture to include automation and control systems (See Figure 1 above).

Communicate with one another. Verhappen adds, “Communication is always key to improving relationships. For example, once folks understand why a required level of reliability is necessary, they gladly will work together to develop truly creative solutions. As a result, both groups gain wisdom that they can reuse for the good of the company.”

Rezabek adds, “For the most part, I think both IT groups should ‘tread lightly’ in each others’ domains, and we should do our best to play well together.”

The bottom line is that plenty can and should be done to integrate plant and enterprise IT responsibilities. Just don’t forget there’s also a need to address process control.

Peter Martin, vice-president of performance measurement and management at Invensys, has a way of making you look at things differently. That’s just what he did when we talked to him about the issues associated with plant and enterprise IT.

Martin believes that for far too long IT, and to a lesser extent I&C departments, have been preoccupied with technology when, in fact, both groups should be working together to create real business value for their company using existing technology investments.

He says business leaders are digging in their heels, declaring, “It’s not about connecting DCS and ERP systems or the latest Microsoft this or that; it’s about running the business better! If you want more money for upgrades and new technology gizmos, build a business case with tangible numbers that you’re willing to stake your job on!”

“I know that sounds harsh,” says Martin, “but in today’s economy, business leaders must put a stake in the ground. They must make their companies behave differently than in the past, or else they—both the executives and the company—won’t exist. That’s where automation professionals can deliver solutions. It’s the motivator for melding IT domains into a unified culture.”

When Martin says automation professionals have an advantage, he’s talking about the control aspect of automation—performing measurements, analyzing those measurements with business objectives in mind, and initiating actions that bring the measurements and objectives into alignment—all in real time, by which Martin means at least daily.

Automation professionals understand control, but probably only in terms of pressure, level, flow, etc. They probably haven’t considered how they might apply their knowledge to implementing business control, but when they do, they’ll find they need better connectivity with business systems. That’s where the automation and IT partnership comes in. To achieve what Martin is suggesting requires:

• Eliminating cultural islands
• Learning to use and express things in business terms (productivity, not availability, reliability or utilization)
• Understanding the processes to be controlled (influencing variables)
• Identifying measurable business goals (set points)
• Establishing critical process indicators (key manipulated variables)

One other major hurdle must be overcome—the self-perception of people working in automation.
More than 25 percent of respondents to an online poll conducted in conjunction with the Control magazine salary survey (June 2006) replied, “We’re more of a set of tools than a profession. I think of myself as an engineer, not an automation professional.”

By definition, professionals are people that have or demonstrate great skills in a particular discipline, are experts in a field, and who are getting paid for what they know.

Answer this: Who in your company knows more about identifying the causes of so-called “uncontrollable variables” than people working in the field of automation?

You are professionals. Stop making excuses and get in the game!