Practical methods for protecting industrial networks

By Jim Montague, Executive Editor

DON'T FREAK out. That’s the main thing. No one ever truly improved their security—let alone their industrial network security—by panicking and overreacting.

Unfortunately, after years of having to do little or nothing to address potential security issues, many network managers and engineers are being jolted into immediate action by visions of hackers and terrorists laying waste to their networks and control applications. Unfortunately again, many are reflexively jumping on quick-fix hardware and software solutions that promise iron-clad protection, but typically deliver inadequate or ineffective results. So, that’s where all the Y2K vendors went!

This brings us to the second main thing. Security requires adaptive intelligence—your intelligence. To engage it, users probably will have to do things they’d rather not do. Besides staying focused, they must thoroughly inventory their whole industrial network, and identify every way data gets in and goes out. And, even worse, they need to create an open, close, ongoing relationship with their IT colleagues.

Yes, there are no magic bullets or miracle cures here. Just the long, slow, unglamorous chipping away that common-sense vigilance always seems to demand, especially to stay ahead of ever-evolving threats. Sure, it’s boring, but it’s more effective than anything else.   

More Connectvity = More Vulnerability
In past years, process control engineers often had to worry more about accidental security breaches typically resulting from non-malicious, well-meaning mistakes by internal users. This situation has changed, mostly because it’s much easier now for even inexperienced hackers to use widely disseminated software to launch many potentially destructive attacks on vulnerable networks (See Figure 1 below).

Software tools used by hackers are becoming more sophisticated and easier to use.

The few experts in the process control security field report that most large, corporate networks are probed and attacked—usually unsuccessfully—on an increasingly regular basis, sometimes three or four times per day. The experts add that virtually all network managers refuse to talk about their security experiences and resulting strategies because they fear it will increase their vulnerability.

The second force pushing process control networks into more vulnerable areas is the quickly multiplying link between process control systems and companies’ enterprise levels and the Internet—usually via Ethernet and wireless technologies. Many of these devices also are far more widely distributed on the plant floor and beyond, and so it’s harder to keep track of them.

“One of the biggest security issues we struggle with is the operating system changes that seem to happen weekly with Microsoft,” says Tony Tenison, control instrumentation manager at Sverdrup Technology, a system integration division of Jacobs Engineering in Tullahoma, Tenn. “The updates require us to be connected, but that makes us more vulnerable. However, if we don’t get the software patches we need, then we have lousy system performance. It’s a real Catch 22. We usually just live without the patch because most of them don’t apply to our operating system. We’ll only go out when we need a specific patch that will definitely improve our performance.”   

Similarly, attacks against specific PLCs reportedly are becoming common because most now have Internet IP addresses and generate clear-text data, which can be monitored by outside entities. In response, several software solutions were developed in the past couple of years that can force a PLC into its administrative mode when an unauthorized device is detected. Unfortunately, this monitoring capability also can enable man-in-the-middle attacks, in which an external device inserts itself between a PLC and an HMI, poisons the HMI’s ARP table, and makes each think it’s seeing the other when they’re really interacting with the intruder. 

“Process control system connections to the Internet and enterprise systems now account for 50% of all control system downtime,” says Rich Clark, information security analyst for Wonderware. “Research by Wurldtech Analytics indicates that, presently, 35% of all control system breaches are due to connectivity to the Internet, and a full 15% of those breaches or similar events can cause downtime. This scares everyone.”

While accidents and internal mischief still account for half of all control system downtime, the other half reportedly consists of external attacks. Of these, Clark reports that 50% are kiddie hackers and 50% are nation states or their agents probing for data gathering or industrial espionage purposes, including some genuinely malicious efforts to bring down applications and facilities.

Though software-based attacks have evolved in recent years, they usually still take the form of denial-of-service attacks or continual-buffer-overflow events. These attacks seek to overwhelm computing systems with numerous unnecessary requests for information or by delivering huge amounts of unneeded data. 

So, the first logical question is, “Why allow any process control system to link to the Internet?” Clark adds there are almost as many reasons for connecting to the Internet as ways to do it. “This is a very slippery slope,” he says.

For example, management and other enterprise users often desperately need up-to-date production data, so they can try to make better decisions that will help their companies compete. Also, system integrators and vendors might need to bring land lines into a plant to monitor equipment, while other users might bring in unauthorized or unchecked USB data storage devices or un-patched laptop computers when doing their jobs.

“We live in a capitalist society, and so we can’t prevent the enterprise level and Internet from accessing data,” says David Teumm, consultant and author of Industrial Network Security, published by ISA. “However, people eventually can find a way to defeat even the most careful security measures. Users can install all the security devices they want, but if they don’t address the human side, then they won’t have security. We just have to put in the most practical security we can, and hope for the best.”

Control vs. IT Perspectives
A third reason why control systems can be more vulnerable to external attacks is that they traditionally have 15-20-year lifecycles, while computer and software lifecycles are measured in months. Because of this time lag, all the resulting legacy equipment is more likely to have older, simpler HMIs and a mish-mash of incompatible software that are more vulnerable to external threats.

“We’ve simply had a lot of traditional neglect of security, so now we have a lot of confusion,” says Clark. “Most users and their companies either don’t have security policies and procedures, or their procedures are far out of date.” Clark adds that this situation triggers two more security problems. First, many IT people still believe that technology can solve any problem, so they just want a widget that will solve all their security issues. Second, control engineers and IT administrators still don’t understand or accept each other.

“It still is a big war in most companies over how to deploy control systems, what resources are available, how networks should be connected, and who owns the machines,” says Clark. “This conflict  exacerbates network vulnerabilities.”

Eric Cosman, engineering solutions architect at Dow Chemical Co. in Midland, Mich., adds, “It’s going to take a long time for standards bodies to develop their consensus-based standards, so don’t wait for them to rescue you. There’s a lot of useful information available to you now. Talk to your own IT and control people, and if they aren’t talking to each other, force them to do so. They must work together because dialog and partnership is the only way for us to handle rapidly changing environments, especially as they relate to security and safety. We need to reduce the time from when Microsoft releases a software patch to when a control systems vendor certifies it for use, and shorten it from weeks or months to days or hours.”

Back to Basics
Once hyperventilation and hand-wringing lose their novelty (and after the control and IT folks are chained together), users can begin to investigate some common-sense security solutions.

To draft useful network security policies and procedures, users must look at all the data their network is moving, why they’re moving it, and who needs to access it. Then, they must decide how to verify if a perceived threat is real, determine who to contact when an event happens, decide how to respond to each type of possible threat and, finally, determine what equipment to shut off.  Next, after policies, procedures, access authorizations, single-point failure and risk analyses, risk mitigation, and a contact tree are in place, the whole infrastructure must be reexamined every quarter, and tested with an actual threat exercise.

“Common-sense security is resisted because it means a big effort,” says Ernie Rakaczky, Invensys’ business development manager for control systems security. “I think plant-floor people know they can’t just do one thing to be secure. They’re realizing that they have to make security a way of life.

“When safety is the issue, everyone watches out for each other, and says, ‘Safety depends on me.’ We have to adopt this same attitude to be successful with cyber security.”

Surprisingly, Ethernet’s recent evolution might help increase network security for Internet and wireless networks. When developers first started implementing Ethernet in industrial networks, many folks were extremely skeptical because it was an office-based technology that wasn’t deterministic, and its hubs could indiscriminately blast data to all nodes on a network. Today, intelligent switches and routers move data via Ethernet only to and from specific locations at pre-determined speeds, while virtual private networks have made it more secure. Intelligent switching, verifying data origins and destinations, and other specifications also can make Internet and wireless communications more secure.

“Security wasn’t as big an issue before because networks were hardwired, proprietary, and even fieldbuses didn’t usually network more than 30-50 devices,” says Larry Komarek, automation product manager, Phoenix Contact. “Now, users can connect hundreds or thousands of devices via Ethernet, so we’re seeing plant-floor networks that are being totally separated from the Internet, or are using commercial IT routers to isolate and analyze network traffic patterns that have the characteristics of someone probing or attempting to attack the network. Protecting a plant in this way requires IT expertise.”

Komarek says some managed switches have added web pages laid out like PLC programs, better graphics, and plain English instructions to show plant-floor users what security they need to use more quickly. He adds that simple, unmanaged switches now have added mechanical locking arrangements, which can block unused ports or lock cable connections. Meanwhile, managed switches have port security that only allows them to talk to assigned addresses, and prevents them from locking onto unassigned connections.

At the next level up, initial switch access can be restricted by passwords or assigned IP addresses. More security is possible by enabling 128-bit encryption between switches. Virtual private networks (VPNs) and tunnels can be used between switches or devices at this level. Also, users can access switches via simple network management protocol (SNMP), and shut down communications between managed switches and the network interface. Finally, the last security level regulates network access via virtual local area networks (VLANs) that isolate traffic and only allow access to certain parts of a network.

In addition, with help from their IT colleagues, control engineers also can turn on IP Sec, which is the secure protocol for Ethernet TCP/IP. Clark says it’s been available for years, but has been mostly unused. IT staffers also can help create VPN tunnels to smart devices and PLCs.

Layers, Tools, and Training
Though there are many different network security methods, most gather around a few basic principles. Perhaps the most significant of these is adopting a network segregation strategy. While suppliers might refer to it as adding layers, rings, or shells, it involves taking devices responsible for controlling a process and parking them behind a second or third firewall, router, or other barrier device, so users can better determine what outside information can reach them (See Figure 2 below).

“This is basically a single point where all the communication paths are controlled, so rules can be imposed about what goes across,” says Tom Good, project engineer at DuPont’s Engineering Group. “You just have to look at all the points of entry, and if you have a firewall at that access point, then you can say there are no dial-in modems or wireless connections allowed beyond it. However, you also have to be careful. Users always are looking for a definitive, prescriptive solution, but they really need to determine security practices that fit the risks and their tolerance policy for their particular process control system. For example, if you’re making some kind of hazardous product, then your security needs will be much different than if you were making distilled water.”

While many security measures are implemented via software, hardware solutions are increasing as well. These can include mechanical locks, dedicated cables and connectors, and even inexpensive biometric devices.

Good adds, however, it’s still most important to train users in safe computing practices, such as making sure that any laptop PCs used outside a facility follow the same security procedures as those used inside it. “Our DCSs have dedicated engineering stations on our process control network, and they usually never connect to our general-purpose LAN,” adds Good. “Even wireless can be nearly as secure as hardwiring. Again, you need a positive authentication of who’s connecting, and you should only allow them access to the devices they need to do their job.”

Despite all the initial confusion, Teumim says he’s optimistic about the future of network security. “People worry that we’ve got 40 committees that haven’t agreed on a standard yet,” he says. “However, what’s happening now is a lot better than having no network security committees and no one caring. We may have too many cooks now, but it’s better than having no cooks, no pot, and no soup.”

[javascriptSnippet]

Standards Stewpot

TO HELP their clients, members, and constituents, many government departments and trade associations have been simultaneously developing guidance and standards for improving network security. Several industry observers say there are presently about 40 government, trade, and corporate organizations developing network security standards, and that 38 of these groups had been unaware of similar projects by the others. Many of them now are trying to coordinate and consolidate their standards work.

Perhaps the largest standards effort is being carried out by the U.S. Dept. of Homeland Security and the National Institute of Standards and Technology with help from Idaho National Laboratories and Sandia National Laboratories, which jointly offer the National SCADA Test Bed to check products for vulnerabilities.

DHS and NIST also have established the Process Control Systems Forum and the Process Control Security Requirements Forum (PCSRF) to gather input on security needs and best practices, which could be included in future security standards.

Other guidelines and standards are being drafted by ISA’s SP99 committee, the North American Electric Reliability Council, the SANS Institute, and the Chemical Sector Cyber Security Program.

DHS and NIST also are affiliated with the U.S. Computer Emergency Readiness Team and its Control System Security Program (CSSP), which lists control systems incidents, and helps users work with suppliers to resolve disputes involving control system vulnerabilities. 

To help all the standards efforts join forces, NIST is compiling all available network security guidelines from the 40 bodies, and reportedly plans to publish them as its 800-53 draft standard in 2007. This coordination is expected to help these organizations decide the security needs they have in common and the methods they can share, and also which aspects of security might be unique to their users and organizations.

For example, NERC’s newly adopted Critical Infrastructure Protection (CIP) standards, CIP-002-1 to 009-1, reportedly can be adopted, altered if needed, and adhered to by users in applications outside NERC’s jurisdiction because they both use computer systems and software in the same way. These commonalities are expected to direct efforts on creating a unified set of network security standards. NERC’s standards cover critical cyber asset identification, security management controls, personnel and training, electronic security perimeters, physical security of critical cyber assets, system security management, incident reporting and response planning, and recovery plans for critical cyber assets.