Building a united front for cybersecurity

You can't stay awake forever. So even though effective cybersecurity demands ceaseless awareness, monitoring and mitigation, you can't go it alone. From the overlapping shields of the ancient Greeks to the shift changes and teamwork needed in today's process industries, we need others we can rely on to spell us and help us succeed.

By now, the basics of cybersecurity should be well known by almost everyone:

• Turn on passwords and antivirus software;
• Separate device and production networks from enterprise/business networks and the Internet with managed Ethernet switches used as firewalls;
• Segment operations into functional sub-networks with more firewalls;
• Enable read-only functions that allow operations to send out data, but prohibit any incoming instructions or commands;
• Adopt and follow appropriate software patching policies;
• Train and retrain staff to follow agreed-upon security procedures;
• Establish regularly scheduled network traffic evaluation using IT-based software tools that can identify, disallow and purge unauthorized probes and intrusions.  

However, because dealing with cybersecurity threats and attacks is a constantly evolving chore, it doesn't have an endpoint or final barrier to hide behind and relax. This also means the steps above are just the beginning of what's needed for effective security. Because time and labor are limited, what's really required for continuous cybersecurity is better and simpler tools, and cooperation among all of a process organization's members, integrators, contractors, suppliers and end users. Heck, the bad guys work in teams and share malware tools worldwide, so why shouldn't the goods guy cooperate to beat them?

Get on the same page

"When we talk about security risks, it's not a matter of when, but rather how one contains and limits the impact of a cybersecurity risk to industrial manufacturing," says Jim Labonty, director of global automation at Pfizer Global Engineering. "Every challenge to devices, applications, computers, networks and physical facilities is serious, and needs to be considered when protecting plants and manufacturing sites. The key takeaway is that no single product, methodology or technology can secure today's manufacturing control system applications, so we need to collectively work together on all aspects, such as patching software and running antivirus programs, to make sure we've established integrated layers of defense."

Labonty reported that a war on automation infrastructures is underway, and that external intrusions and attacks have been ramping up for the past 10 years. However, he added that control systems can no longer rely on their historically physical isolation because so many now have links to higher-level enterprise systems and the Internet to get useful data out. Unfortunately, this creates security vulnerabilities that must be managed.     

"Pfizer isn't perfect when it comes to cybersecurity, but we're working with our plant sites to establish these secure layers," explains Labonty. "We're finding that they have different levels of security capabilities, but we also know this is continuous process for everyone. This is because intrusions and cyber-attacks are growing increasingly sophisticated. In fact, the number of attempted cyber-attacks on most manufacturing site—including Pfizer's—is now in the millions per day, so we've got to get cybersecurity infrastructures in place from our global networks down to the plant floor. Our initial cybersecurity designs were usually two network interface cards (NIC), Ethernet and servers, but we've been updating them to better designs."

[javascriptSnippet]

And, as if existing security situations weren't dire enough, Labonty reports traditional hackers are increasingly joined by nation-states bankrolling teams of attackers breaking into corporate networks down to their lowest levels, mostly to discredit and disrupt their brands.

"Control systems must establish defense in depth, but they can also look at sending network logs and data back up to users for inspection," adds Labonty. "This can be very helpful because it lets users see if anything has changed or gone wrong at the control level, which is a huge advantage. Defense in depth strategies can also define authorized traffic, so at Pfizer, we use a series of firewalls as our network goes down to the controls level, where there are more secure zones. Firewalls aren't too costly, and they can pay back quickly, and report their logs back, too. We're also using Splunk software to analyze network traffic patterns, which gives good indications when something is trying to transgress and a proactive indicator of what to investigate."

Security joins control, operations

Beyond gathering and coordinating available security tools and related players, some users emphasize making cybersecurity part of their regular data acquisition, quality tracking, process safety and other efforts.

Dan Stauft, corporate engineer at Sugar Creek, reports it's celebrating its 30th year of manufacturing mostly private-label bacon and other food products at its 420,000-square-foot facility and at six other plants it's integrating. "Previously, we only did data collection for our machines, so we've started exploring Inductive Automation's Ignition SCADA software for use with our water treatment and refrigeration applications, and we're seeking to link them with our MES system," says Stauft. "We also do manual quality assurance logs to prevent recalls, so we're also starting to put in place better temperature monitoring of our refrigerators and freezers. This includes adding automatic alarms and reporting, so we can identify approaching temperature thresholds before there's a problem and we risk spoiling 20,000 pounds of pork; respond proactively instead of reactively; and make it easy for users to extract data without needing a master's degree in SQL programming."

Stauft adds that Sugar Creek also does product tracking and tracing, and that it's trying to plug Ignition software into this application, but do it securely. As a result, it employs Cisco's Converged Plantwide Ethernet (CPwE) architecture and topology (Figure 1).

"Our two-year-old plant in Indiana is a Cisco show plant for security," says Stauft. "The applications and equipment are air-gapped, and no one is allowed in through the VPN to the manufacturing zone. Instead, we run virtual desktop interfaces (VDI) with VMware software and servers. They sit in the network demilitarized zone (DMZ) that's configured for our suppliers to access their machines. They allow remote logins at the VPN, but make users go to the VDI to see machines and applications. It's like logging on through a remote desktop, but only VDI can talk to the machines. Even the vendor doesn't have direct access here, and can't push or pull files because they're all located at the DMZ on the devices. Each vendor and machine gets its own subnet, and the Ignition server is the only one that can access all of them. To access a PLC, for example, the vendor uses VPN to open a remote desktop session on a VDI. Each VDI has whatever software the vendor needs loaded, and only has access to the VLANs assigned to the vendors devices."

[sidebar id =3]

Sven Schrecker, chief architect of IoT Security Solutions at Intel and co-chair of the Security Working Group at the Industrial Internet Consortium (IIC), adds that, "Security is one of five characteristics that support Industrial Internet of Things (IIoT) trustworthiness. The others are safety, privacy, resilience and reliability. Together, they create the trustworthiness that IIoT needs to protect against system faults, environmental disruptions, human errors and cyber-attacks. We need a new, comprehensive adoption model for trustworthiness as the basis for industrial adoption of IIoT. Then we need to look at all environments from a security perspective, and leverage trustworthiness to manage risk and increase the likelihood of correct business decisions. Security can't be something we do just to do it or for compliance."

To aid these efforts, IIC recently published a 173-page guide, "Industrial Internet of Things Volume G4: Security Framework." The Industrial Internet Security Framework (IISF) offers a security model and policy built in conjunction with the Industrial Internet Reference Architecture (IIRA). IISF has a data protection layer with several security building blocks and techniques for IIoT, including security configuration and management, security monitoring and analysis, communications and connectivity protection, and plant protection that includes edge devices and the cloud.

"We need chips, boards and software with security built in from the beginning, and we need them attested to the right level of security from the top down," said Schrecker. "Owner/operators also need to demand better security and tell system builders and component builders to assert trust in their systems. End users can also assess security levels.  Performing all these tasks is the only way to get consistent security into the IIoT. Also, we can't just be secure at the edge, and think we're secure overall. We need end-to-end security based on comprehensive models and policies. Each part of an application needs to protect itself, whether it's at the edge, on the network were in the cloud."

Utility, community network secured

Naturally, greater cybersecurity is needed as more process applications form links with business-level networks, external data sources like the Internet, and more widely distributed remote service providers and communities.

One of the latter is Clarksville Light & Water (CLW) in Arkansas, which is a 103-year-old, municipally owned utility that provides electricity, water and wastewater services at not-for-profit rates to 4,500 retail customers and just under 10,000 residents, as well as wholesale water to eight nearby towns and local water districts with about 18,000 more customers. The utility spent much of 2015-16 enhancing its municipal infrastructure with 16.7 miles of 288-strand fiber optic cable installed in redundant loops throughout the city to avoid single points of failure. This project included implementing Ignition SCADA software and Bedrock Automation's control system for cyber-secure SCADA RTU monitoring and control of CLW's electric, water and wastewater facilities (Figure 2).

This new, secure network is expected to potentially save CLW up to $2 million over the next five years by improving control, monitoring and security of the city's electric grid and water treatment facilities. Besides reducing purchased power losses, the fiber optic network is expected to allow CLW to provide broadband Internet to residents, and keep another $810,000 per year in the community.

[sidebar id =4]

"Besides dealing with legacy applications, controls and networks, there's also a lot of bureaucracy to deal with in municipalities and corporations when you're trying to implement cybersecurity," says John Lester, CLW general manager. "You have to explain how to do it to supervisors and managers, and convince them of the need for it. There are also many political and social obstacles, so you also need to communicate with everyone upfront, and let them know what's coming.

"The fiber optic network gave us a way to tap new functions for our remote operations. And, when we learned that choosing Bedrock as our RTU system would mean cybersecurity was already built in, we saw it as a cost-effective way to reduce our security risk while also addressing looming North American Electric Reliability Corp. Critical Infrastructure Protection (NERC-CIP) compliance requirements."

Dee Brown, PE, principal at Brown Engineers LLC in Little Rock, Ark., which provided electrical, mechanical and system engineering services for CLW's project, adds that, "The operators know about firewalls and Industrial Control Systems Cyber Emergency Response Team. But if there's a phishing attack and something gets in, how do you protect against it, and how can you do it with a limited time and budget? Now we have solutions for that.

"We wanted each substation RTU to have enough horsepower to aggregate all power meter data and protective relay data for sequence of event recording. We also considered future development needs for power management techniques that support demand management and load-shedding controls. Bedrock controllers provide those features in an integrated development environment (IDE) that uses IEC 61131-3 programming and has cybersecurity embedded at the hardware level. These days, cyber attacks are not if, but when. However, if we can do cybersecurity like this with Bedrock controllers, then we think everyone can and should be doing it. For us, we're working in the same PLC-type environment as always, but now we have to do better than protecting devices in bubbles at the edge of the network. The elegance of the Bedrock system is that the supply chain manufacturing process embeds the security keys to ensure the highest levels of hardware and software authentication.”

For instance, extending the new controller/SCADA deployment to the grid offered a simple, effective way to add secure control and monitoring of remote assets. At each of CLW’s substations, a Bedrock controller directs and protects data transfers along the new citywide fiber ring. This design allows all five Bedrock controllers to connect directly to the fiber optic cable without added copper connectors. Brown reports they've been up and running since June 9 with no problems. With this infrastructure established, CLW plans to use automated circuit switching, as well as monitoring and load balancing, on the local grid, which will let it realize demand-side management for the electric utility. CLW expects to improve overall reliability, shorten response times, and reduce power supply costs.

In addition, Brown Engineers developed an application that extended the communication libraries provided by the Bedrock IDE to allow communication with the existing protective relays. "Common design elements allowed us to connect to legacy protective relays and power meters," adds Brown. "We can now see daily weather and temperatures, and show how they're affecting the electric utility's load. We'd never had this onscreen before. We also have screen maps and operating screens for each of the four substations."

To help users monitor and handle the sheer number and variety of possible cyber probes, intrusions and attacks, several developers have been building solutions that can not only give them a heads up, but can also aid in preventing, responding to and mitigating many of the most common and severe threats.

"We still need to raise awareness levels about cybersecurity because many users continue to believe they're isolated, don't see that cyber-attacks can happen to them, and aren't conscious of the possible attack vectors," says Robert Alston, cybersecurity leader at Honeywell Process Solutions.  

As a result, Honeywell launched its Industrial Cyber Security Risk Manager real-time data collection and analytics software platform that uses proprietary algorithms to continuously monitor process control systems for cybersecurity risks, vulnerabilities and gaps. It consolidates complex, site-wide, cyber-threat and vulnerability data into one view for better visibility and improved decision-making; extends users' ability to stay ahead of cyber-threats in ways not previously possible; runs on non-Honeywell control automation systems; and integrates with enterprise security platforms such as security information and event management systems (SIEMs) as well as leading network and endpoint security products.

[sidebar id =5]

"Industrial Cyber Security Risk Manager combines software and hardware to provide a total view of a process operation's status, and then shows its users what they need to do and where for cybersecurity, such as what patches they need," adds Alston. "It also does analytics about what vulnerabilities are going on in adjacent systems, and then our Managed Services Group can help users acquire the patches or other cybersecurity solutions they need. We're also developing a real-time process control monitor with Palo Alto Networks for checking network traffic and detecting anomalies, and next year we're going to release Secure Media eXchange (SMX), which is a solution for securing data transfers via USB devices."

Barak Perelman, CEO of Indegy, reports that downtime due to cybersecurity incidents doesn't just come from nation-states or other premeditated hackers, but can also come from unintentional errors by otherwise well meaning personnel or third parties. He added this is why it's vital to get OT and IT personnel to cooperate on cybersecurity best practices.

"Indegy can help with these cybersecurity efforts because our main concept is providing visibility into ICSs, and while there are dozens and hundreds of cybersecurity products for IT, there are few on the ICS side," explains Perelman. "We can view all that's happening on an ICS, detect any changes whether they're malicious or unintentional, and provide alerts and an audit trail. Our solution is 100% visible, so users can track what's happening to their protocol, detect malware, and decide the best way to recover from it."

Earlier this year, Indegy launched its first cybersecurity platform for providing comprehensive visibility into OT and control networks to identify threats that put ICSs at risk. Its solution detects logic changes to controllers that manage power grids, gas pipelines and manufacturing facilities, and protects against cyber-attacks, insider threats and unauthorized modifications. Its two main tools include a control network inspection (CNI) tool for analyzing communications and protocols, and a device for communicating with the controllers in a given network to extract a baseline snapshot that can be used to indentify subsequent changes or tampering.

"We put Indegy platform software between PLCs and managed Ethernet switches because visibility of the networks and any configuration changes are the first step to cybersecurity," adds Perelman. "If any changes have happened, Indegy can help show how many, what they are, and where they are. There first stages of cyber attacks are reconnaissance and data gathering. However, once this activity and other unauthorized communications become visible, then users can dive in, seek malware, and recover more quickly."  

Likewise, Claroty released on Sept. 13 its software-based platform for providing visibility into industrial control systems (ICS), and ensuring security and reliability for them and their networks. The company reports that Claroty employs high-fidelity models and advanced algorithms to monitor ICS communications and provide security and process integrity alerts, which enable users to discover hidden problems in OT networks, protect critical systems against cyber-threats, and fix issues that impact process integrity and performance.

"We go in passively, discover all of a network's assets, watch and figure out how they're talking to each other, and build a detailed model of those communications," says Patrick McBride, senior vice president of marketing at Claroty. "This lets us detect any anomalies, potential malware and other process integrity issues."

Hardware aids software

Ironically, while many recent cybersecurity solutions are founded and deployed in flexible software formats, they're also using hardware to provide added protection against intrusions and attacks.

"Users can harden devices and implement edge security control by taking software from those devices, virtualizing it, and putting it on their gateways or servers," explains Intel's Schrecker. "This puts the soul of a device like a PLC in its gateway, so users can get data from virtual sensors. The traditional way of protecting end devices is putting a software agent in operating systems, but we say it's better to put a compressor chip in the end device, so you can have operations on one chip and security on another chip."

A security chip can monitor and manage all security tasks, enforce firewall functions, store identity information, mutually authenticate devices and users, and authorize network traffic. A security chip can also be defined as the only device that's allowed to talk to the outside, so all operations chip communications go through the security chip, and follow its security models and policies, much like a network white list.

"All of this gives security a place to run," adds Schrecker. "Then you can carry out more sophisticated security management, reactively and proactively update devices as needed, and pull security data including metrics and KPIs for better security monitoring and analytics."

For instance, Emerson Automation Solutions launched on Oct. 25 its Secure First Mile program for securely connecting operations with IT and cloud-based services by using architectural approaches and designs, security services, and flexible servers, gateways and data diodes, which work together to make certain operations technology (OT) systems can be easily and securely connected to Internet-based applications.[sidebar id =6]

"Secure First Mile is located close to where process data is generated, and converts sensor and other production-level information into secure data for the Internet and cloud level," says Claudio Fayad, technology vice president of Process Systems and Solutions at Emerson. "Secure First Mile uses the data diode method to simplify protection of inbound communication. Data diode uses two transceivers with a standard RJ45 cable and bidirectional input on the field gateway/OT protocol side, and a fiber or RJ45 cable and bidirectional input on the edge gateway/IoT protocol side. The data diode model provides extra security because it creates a physically unidirectional network that only lets data out, but has no physical connection for letting data in. This enables production data to go the edge gateway for conversion, so it can be used by IoT protocols and made available in the cloud, but physically disables the inbound path and creates an air gap that prevents any communications from the outside."

Likewise, FDT Group released an annex to the FDT standard for OPC Unified Architecture (OPC UA) on Nov. 23, which will enable sensor-to-cloud, enterprise-wide connectivity with on-the-wire security in industrial control systems used in process, hybrid and factory automation. FDT/OPC UA annex will allow automation suppliers with FDT Frame Applications (FDT/Frames) embedded in their DCS, asset management system, PLC or other system to include an OPC UA server in an application that's accessible from any OPC UA client application.

Share and spread awareness

Following deployment of all available technical security fixes and internal policies and training, Pfizer's Labonty stresses that manufacturers, system integrators, suppliers and contractors must keep on sharing their cybersecurity know-how with all related parties, so they can develop and present a unified response to probes, intrusions, threats and attacks.

"Awareness by everyone is the key because we're only as secure as our weakest link," said Labonty. "Our older networks just had one firewall between IT and the production levels below, but behind this castle-and-drawbridge, there was a free-for-all of data going everywhere. So, we've rigorously added another layer with secure zones and cells divided by protecting each business asset from the others. These secure areas are divided by purpose-built firewall gateways, such as Rockwell Automation's recently released Stratix 5950. We also segmented a lot of older equipment away from our newer systems and devices."

Similar to physical networks, Labonty concluded that cybersecurity also requires users to decide on and establish clear demarcation lines between their site automation teams and their IT counterparts. “It’s good for security to establish clear roles and responsibilities, and it also helps when different players need to talk to each other," added Labonty. "This demarcation is also important because Pfizer outsources a lot of IT, and they're not familiar with our individual sites. So, we definitely don't want them trying to manage any production because they don't know the ramifications of their actions." 

Finally, personnel and organizational issues like these are the most important cybersecurity issues for process control and automation users and suppliers to solve, according to Labonty. "The easiest and most popular ingress for cyber attacks is spear phishing, which tricks people into opening emails and click on attachments that download malware," he said. "So educating workforces on policies and procedures to protect against it and other threats is also crucial, too."