Traffic cops keep watch
Of course, the ultimate aim of any cybersecurity effort is the same as any other plant-floor initiative from basic loop control to advanced process optimization and safety—keep the application running as efficiently and profitably as possible. However, because there's no "set it and forget it" with cybersecurity due to constantly evolving probes and threats, a secure network and the communications traffic on it must be constantly examined for anomalous performance that could indicate unauthorized and possibly malicious activity. Earlier networking monitoring tools like IT-based simple network management protocol (SNMP) and related derivatives have given way in recent years to passive-monitoring software like SIEM that are less likely to hinder operations.
"It's OK to use SNMP, but you have to be careful because we can't have active network scans, which will lock up many legacy devices on most plant floors," says GPA's McNeil. "We try to use SIEM software wherever possible because it's passive; comes in many flavors; has antivirus and anti-malware capabilities on top; uses mature signatures and rules-based authentication; and can incorporate machine learning and artificial intelligence functions based on behaviors. This is really the future of cybersecurity because behavior-based software monitors internal data traffic—where it's from and where it's going—but it adds devices on a network to establish its normal, permitted traffic pattern, which lets it identify anomalous traffic and deliver alerts."
McNeil adds that GPA often uses SIEM software from AlienVault, works with machine-learning provider Darktrace, and can integrate these tools into users' networks. "We work with clients to examine their risks; establish their security policies and procedures; and set up education, training and enforcement."
Enlisting further help from the IT side, Matrix's McKarns reports, "Microsoft Active Directory is used to enforce security and access control policies on domain joined objects. It gives administrators one pane of glass from which to view and manage permissions for all users in a control system. It's the same tool that's been used in corporate enterprise networks for more than 20 years, and now administrators on manufacturing networks can centralize monitoring and management of their users, PCs and network devices, and apply policies to those users and devices allowing greater, more centralized control. Typically, joining machines to a domain wasn't done on plant floors in the past, but it's becoming much more common in new deployments and in situations where IT has a stronger influence."
McKarns explains, if domain services are in place that restrict permissible actions, along with good firewall and intrusion detection policies, many virus and ransomware attacks wouldn't be possible. "These threats are why ISA and ICS-CERT put out guidance that it's not a good idea to connect plant control system PCs directly to the corporate network, and recommend using firewalls and DMZs between layered networks," he adds. "Another element of defense-in-depth is active monitoring of network traffic for anomalies and possible intrusions, and being able to accordingly. The next step is finding the right mechanism for intrusion prevention. Prevention requires an automatic, heuristic software tool, which includes a learning capability and understands typical network traffic; establishes a baseline; red flags unexpected traffic; provides alerts and alarms; and can disallow the flow of unauthorized traffic through key security points. Intrusion prevention systems must be configured and deployed carefully, so they don't block critical traffic."
Matrix's Lycans adds, "Evaluating network traffic was originally more straightforward, but outputs were not always clearly formatted, there was a lot of text, and it could be hard to understand where links were and what their status was. Traffic evaluation tools are more graphical now, and present maps of devices, links and switches. We've used IntraVue for many years, and we also use SolarWinds software that displays the corporate enterprise."
McKarns adds, "Today's network traffic scans aren’t just visualized better, they also benefit from being dynamic network scans in real-time that can better detect intrusions. However, where it used to be easier to scan many switches because their settings were open and unsecured, it's more difficult now because more of them have built-in security requiring a greater amount of configuration."
From defense to anticipation
[sidebar id =1]help users better understand how cybersecurity works and protect their processes, Sid Snitkin, VP and GM of enterprise advisory services at ARC Advisory Group, advises users to follow its Industrial Cybersecurity Maturity Model, which moves from reducing the likelihood of intrusions and attacks to reducing their impact (Figure 2). Its levels include:
- Secure with physical security, asset inventory, device hardening and patch management;
- Defend with unidirectional gateways, demilitarized zones (DMZ), firewalls, anti-malware and access control;
- Contain with zone firewalls, ICS device firewalls and whitelisting;
- Manage with security information and event management (SIEM) software, and incident management solutions; and
- Anticipate with anomaly and breach detection methods, and threat intelligence.
"Security investments should have specific objectives, and lower-level goals should be achieved before advancing to higher-level goals," says Snitkin. Many users and organizations discover they've bought too much cybersecurity technology, and find they've got a lot more than they have the resources to manage."
Eric Knapp, chief cybersecurity engineer at Honeywell Process Solutions, adds that, "There are many ways you can seek to better understand security events and how vulnerable you are to the latest threats. At the most simplistic level, think about your security across three major categories: people, process, and technology. Then take any of the latest incidents, such as Wanna Cry or Petya, and scenario plan how they might impact those categories. You’ll quickly uncover gaps that you can address.
For example, we've seen that many companies that had hired managed security services teams to keep security patches updated were protected against WannaCry and experienced no negative impact from the cyber attack. Those companies had recognized they didn't have the personnel to access and qualify patches, administer updates, and keep them regularly updated, so they hired in service teams to address this gap. Breaking security into categories is a practical way to tackle issues specific to your plant, and help ensure you aren't over-investing in the wrong areas.
"Another way to understand how security incidents might affect your industrial company is to leverage technical solutions to automate visibility and control over your PCN risks. You can run specialized software that calculates your risk and rapidly visualizes it, for example. Honeywell has the Industrial Cyber Security Risk Manager that measures, monitors and prioritizes risk levels across your operational assets, then offers simple guidance so teams can act on tasks that lower risk."
Bake in, don't bolt on
Yet another major suggestion emerging from sharing on cybersecurity from many players is their call for security software and capabilities to be added during design and construction of industrial components, instead of seeking to add security functions to existing devices and systems.
For example, Vista Irrigation District near San Diego, Calif., recently upgraded its SCADA system, I/O modules and networking from serial to Ethernet—and improved its security posture at the same time—with help from the same two firms that installed them more than 20 years ago, namely Opto 22 and system integrator IDAC West in Sam Marcos, Calif. The district moves more than 5 billion gallons of water per year for its 127,000 customers through more than 435 miles of pipes linking more than 30 remote sites, including 12 reservoirs, source water connection points, pump stations and flow control facilities.
“The distributed intelligence of Opto 22's SNAP PAC let us prototype, test and rollout upgrades to one remote site at a time,” says Alan LeVezu, engineering manager at IDAC West. “We were able to run existing and new systems in parallel, and cut over to the new system when we were ready. We also used IDAC's research and development test center to design and prototype new features of the updated system, which let us verify that the new controllers, I/O, and updated control software were running as intended before cutting over."
Opto 22 reports that dual, independent, 10/100 Mbps Ethernet interfaces on its SNAP PAC controllers connect to separate IP subnets, which improve cybersecurity from the outset by isolating the control and business sides of an overall network. SNAP PAC also has an optional 802.11 a/b/g wireless interface with built-in WPA2-AES security. SNAP PAC also has Ethernet traffic monitoring that allow LeVezu to display network performance statistics between Vista’s headquarters and remote sites directly on an HMI for increased visibility that gives the district near real-time situational awareness at all sites.
"SNAP PAC lets us perform process control at each individual site,” adds LeVezu. “Because each controller and brain in SNAP PAC is intelligent, we can configure setpoints directly at each site. So, if the network goes down, those sites can continue to operate autonomously. With built-in Ethernet already on the controller, we don’t need to buy added Ethernet cards or upgrade the controller to a higher-priced model to add networking capabilities”
Tony Baker, cybersecurity portfolio manager at Rockwell Automation, adds that, "We're trying to take an embedded approach by designing cybersecurity into products, in addition to establishing a product security office, which identifies inherent risks, consults product teams as to how they can mitigate those risks, conducts penetration testing, and completes validations and verification testing for implementation. In short, cybersecurity is taking on a product quality assurance role, and we have to prove that competency."
Down to the ground
Beyond incorporating security into controls and networks when they're first designed and built, several experts warn that process controls systems remain especially vulnerable at the lowest levels of the Purdue reference model, primarily Levels 0 and 1, where sensors and other basic instruments are located. As more of these modules gain microprocessors and Ethernet connections, they can also be pathways for probes, intrusions and attacks, which can be as subtle as communicating incorrect readings with potentially disastrous result if they're misinterpreted or remain unchecked.
"The policy of setting sensors to 'fail as-is,' 'fail upscale' or 'fail downscale' is commonly done. However, if safety systems are involved, the approach of setting sensors to a fixed value can cause a loss of safety," states Joe Weiss, ICS cybersecurity expert and author of Control's "Unfettered" blog in his Aug. 18 post, "Insecure process sensors can create safety, security and resilience vulnerabilities." "An inherent assumption is that sensor values are correct even though process sensors have neither authentication nor security. Cyber concerns with sensors include cyber-vulnerable protocols and smart transmitter functionality that leads to cyber vulnerabilities. Currently, ICS cybersecurity assumes process sensor input is correct. If sensor values are incorrect either because of unintentional issues, such as sensor drift, miscalibration, etc. or by sensors being compromised by a cyber attacks, then resilience, safety and security can be defeated.
"Following a rigorous approach to understanding complex system behavior over the full lifecycle and under numerous threat scenarios will expose such risks and support mitigation decisions. It's also essential to consider a review and reexamination of failure modes in many of our vital infrastructure systems."
To combat the threat of spurious sensor signals, Eddie Habibi, CEO and founder of PAS Inc. reports that engineers and managers in the process industries must conduct an awareness campaign that brings cybersecurity even further into the realm of process safety, and go beyond network traffic monitoring and intrusion detection by adopting complementary security controls that address the lowest control system levels. Beyond traditional security controls, PAS advocates continuous monitoring of field sensors and actuators (Level 0 devices) using multivariate process behavior recognition technologies to identify anomalies.
"Foundational ICS cybersecurity must include device inventories, vulnerability assessments, configuration baselines, change management and recovery procedures," says Habibi. "Our Cyber Integrity software mitigates Level 0 and 1 vulnerability risk and engineering mistakes by identifying and inventorying the user's whole control system, including its field instruments, I/O points, valves, I/O cards, control loops, and its HMIs, historians and servers. Once it captures the configurations of all network and OT devices, we have a baseline that can be monitored for any unauthorized changes."
John Cusimano, industrial cybersecurity director at aeSolutions, adds that, "Previously, people had to be convinced to address cybersecurity. Now, they want to know how to get started. The market is a lot more sophisticated now that many users already have some cybersecurity in place and are trying to improve it. However, even though many users write security policies and audit their facilities, we always discover vulnerabilities when we perform assessments in the field, such as unsecure TCP ports. Typically, we find there's good segmentation from the business network to the process control network (PCN), but not a lot of segmentation within the PCN. I'd estimate that only about 25% of PCNs are segmented into security zones as recommended by ISA and other industry standards and best practices."
Located in Greenville, S.C, aeSolutions is a consulting, engineering and CSIA-member system integrator specializing in process safety, SIS and industrial cybersecurity. It reports that its aeCyberPHA methodology is a practical application of the ISA 62443 cyber risk assessment requirements, which link realistic threat scenarios with known vulnerabilities and existing countermeasures, and couple them with credible consequences from the process hazards analysis (PHA) to determine cyber risks.
Securing big migrations
Because large process applications have increasing network connections and growing potential vulnerabilities—as well as brand-name recognition that makes them targets—engineers and integrators supporting them must be even more vigilant in their cybersecurity efforts.
"The cybersecurity arena has grown along with expanding connectivity to give dashboards to management, and we've run into it more often as we do many process system migrations mostly in large chemical and pharmaceutical facilities and some in power distribution and food and beverage applications," says Robbie Peoples, integration manager at Cross Integrated Systems Group, a CSIA-member system integrator and one of six divisions at Cross Co. in Knoxville, Tenn. "Cybersecurity is becoming more of a concern because we're still running into PCs running Windows NT, which are very vulnerable because the software is obsolete and no longer supported. Some users report they have a firewall, but we find it would be a stepping stone for a hacker."
[sidebar id =2]
Peoples reports that most of Cross's integration work is on large DCSs with some PLCs, and when they do a migration assessment, it typically includes a cybersecurity assessment for external and internal problems. "We look at the legacy equipment and walk the field, which is essential because 95% of people have an architectural diagram, but it's not accurate with everything they've added since it was created or last updated. For example, when there are supposed to be three wires and there turn out to be five, we don't know what the other two are for."
Beyond a thorough walkdown, Peoples reports that Cross is using open-source Nmap scanning software, which interrogates a network and lists all of the IP devices, hosts and services on it. "We cross-reference the result of the scan with the architectural diagram, which lets us identify and investigate any unknown addresses," he explains. "We always find an item that the client doesn't know what it is, such as an RJ-45 plug-in jumper between switches or a patch panel that someone has added underneath the network. We also scan equipment items on the network to find open ports. For example, an application may be running Wonderware I/O server and an associated OPC server in the background to make the I/O available, but this means opening virtual ports that an be vulnerable to something getting in."
Once it's identified a network's vulnerabilities and risks, Cross generates recommendations. "For instance, we'll check if anyone is using a device, Nmap will identify its ports, and if they aren't being used, we'll recommend shutting them down," adds Peoples. "In addition, just as corporate uses Wireshark open-source protocol analyzer to scan its networks, we've also been using Indegy's hardware appliance to recognize and interrogate our process control networks."
IT + OT = "securiTy"
Another consensus recommendation from users, system integrators and suppliers is that bringing together IT and OT departments and personnel can be a huge help in improving cybersecurity for process application and their overall organizations.
"We do many assessment interviews on our migration projects, and when we ask the IT guys about ICS security, they say 'we do the patches, but we can't go on the critical control network.' Meanwhile, the controls guys say 'we have a router, but it's not IT secure,' " explains Cross's Peoples. "Consequently, sometimes no one is looking at security and there is no policy. In fact, we find that 90% of end users don't have a person dedicated specifically to cybersecurity. This is because many people are coming into and out of process control jobs every five years; Baby Boomers are quickly retiring; their plant network is usually a patchwork quilt; and requirements for doing the security job right will have changed a dozen times over the past 10 years. As a result, no one is taking care of security."
To improve security by helping the OT and IT sides collaborate, Peoples reports that Cross recommends that each user establish standard security procedures, including standard requirements for their network interfaces. "This includes deciding how will the plant connect and what will it communicate? We also suggest doing an audit every six months, which means scanning the entire network, walking it, and tracing down all devices. A third-party audit should also be done every year, which may recommend standard security procedures, address physical security issues, or advise against accessing Windows software from operating stations. We also review policies, such as not allow operators to access engineering codes, clearly labeling cables, and having separate network segments for I/O, controls and servers for clients.
[Peoples has two recent cybersecurity blog posts: "How vulnerable is your ICS?" and "Four steps to secure your automation system."]
"There's greater integration between IT and OT, but the big roadblock right now is internal politics," says Mariam Coladonato, product marketing specialist for networking and security at Phoenix Contact. "IT is looking to integrate with OT, and there's less of a line now where OT and and IT starts. However, IT can also be an obstacle to the proactive monitoring and live reporting that OT and management want, and OT can also block integration when it doesn't want what it perceives will be disruptions on the plant floor."
To iron out some of these snags, Coladonato adds that players on both sides must be encouraged to coordinate their efforts more closely, and that Phoenix Contact is planning to help by developing network-aware firewalls that should reduce their usual frustrations. "We looking at integrating our mGuard devices with Splunk, Silent Defense Skybox and other networking monitoring software, which can connect to a central database for proactive monitoring, but also perform intrusion detection and protection."
Pair ups for services
Because undertaking and maintaining a cybersecurity program requires ongoing labor and resources, many integrator and suppliers are partnering with cybersecurity experts and start-up companies to gain the capabilities they need to serve their users.
For example, Metso recently halved the cost of a large advanced process control (APC) project with help from Skkynet's SkkyHub real-time, cloud-based data collection and distribution system. This combination enabled multiple engineers to work in parallel on an exact replica of the live ore processing plant in the Middle East and its 25,000 active tags, while they built its GUI, sensors and data modeling/analysis functions in France. Metso had previously integrated Skkynet's DataHub web-based HMI with its OCS-4D APC software for mining, and adding SkkyHub allowed remote monitoring.
"The DCS used for regulatory control of the ore processing system exposed 80,000 variables as OPC tags through an ABB OPC server," says Bob MvIlvride, communications director at Skkynet. "Metso’s APC optimization layer required access to about 25,000 of those tags to power expert logic in OCS-4D and the HMI. To get live data back to Olivet, France, the engineers configured OCS-4D DataLink at the ore plant to make a secure, outbound tunneling connection to SkkyHub. Back at Metso's office, its development team made a similar, secure outbound connection from their local copy of OCS-4D DataLink to receive live data in real time, and feed it to their OCS-4D system. This produced the exact replica of the entire ore plant, which was mirrored and tunneled from the Middle East to France, updating continuously in real time. Unlike a typical project, where developers have to take turns using the system data, this unique arrangement allowed each of them to work independently on their parts."
McIlvride adds that SkkyHub's tunneling ability lets users see which side has started a tunnel, and only allows its firewall to be opened from the user's side. SkkyHub can also be added to a DMZ to help protect plant and corporate layers because it can run without opening a firewall. It also compiles with the OPC UA client/server requirement for maintaining a constant "chain of authority" for data as it moves through Skkynet's Secure Middleware from hubs on equipment to outbound connections and clients.
Likewise, Emerson Automation Solutions recently launched its SIEM appliance that runs a tailored version of McAfee's SIEM software in a physical box or virtually, according to Alexandre Peixoto, Delta V marketing manager for cybersecurity. It also released its Network Security Monitor appliance at the same time because it can gather and compile data, and delivers it to the SIEM system. "These solutions are letting us extend SIEM to the process control side, but it also uses port mirroring, so data can only go one way up to the security monitor," says Peixoto.
In addition, Panacea Technologies Inc., a CSIA-member system integrator in Montgomeryville, Pa., recently launched its Update Manger software that automates the process of deploying Microsoft patches on industrial networks by allowing only automation vendor-approved patches to be deployed on appropriate computers. It supports most major automation platforms, and future releases expanding the support base are planned. "Panacea Update Manager will decrease downtime caused by deploying untested and unapproved patches that break automation systems, and decrease overall engineering and validation efforts associated with rolling out patches for automation infrastructures," says Will Aja, customer operations VP at Panacea.
Also, Honeywell recently launched its Secure Media Exchange (SMX) hardware that lets users examine USB devices for approved use in their facilities. USB drives are plugged into an SMX Intelligence Gateway, which analyzes files on the device using a variety of techniques included with Honeywell’s Advanced Threat Intelligence Exchange (ATIX) secure, hybrid-cloud threat analysis service.
Some other recent partnerships between process control suppliers and cybersecurity providers and startups include:
- Emerson joined with Intel Security, recently renamed McAfee, to better secure Emerson's DeltaV DCS.
- Honeywell acquired Nextnine, and added it to Honeywell's Industrial Cybersecurity Group to offer multi-vendor, multi-site, secure remote access, monitoring and support.
- Rockwell Automation added Claroty as an Encompass partner, so they can combine network monitoring software solutions and services as packaged security offerings.
- Schneider Electric is also working jointly with Claroty, which will enable it to integrate Claroty's virtual appliance scanning software as part of its products.
- Siemens and Tenable Inc. partnered to Industrial Security from Tenable, which is an asset discovery and vulnerability management tool that Siemens will deliver as a service.
