One defining characteristic about cybersecurity is that each new capability and gain achieved builds on those that came before—which makes it even more crucial to start with the basics and do them right.
“We’re still seeing many clients who aren’t adopting networks segmentation to the level that we like to see, so we’re still doing a lot of blocking and tackling,” says Scott Christensen, cybersecurity practice director at GrayMatter, a technology solutions consultant in Pittsburgh, Pa., and a certified member of the Control System Integrators Association (CSIA). “This is changing priorities for more users because they’re putting visibility into their systems first, asking what assets they own, what programmable logic controllers (PLC) do they have where, and how can they add value by operating more securely?”
Get your subscription to Control's tri-weekly newsletter.
GrayMatter works in many critical infrastructure verticals, including water/wastewater, and oil and gas. Over the years, it’s helped about 1,500 water utilities implement their PLCs, SCADA/HMI systems, control panels and predictive analytics. However, many are woefully underfunded and understaffed when it comes to cybersecurity.
“The discrepancy for water/wastewater is that many utilities have tried to close their cybersecurity gaps, and have their regulators and governments for guidance, but they’re not getting it so far,” adds Christensen. “The U.S. Environmental Protection Agency (EPA) and the American Waterworks Association (AWWA) have provided disjointed messaging and communications, while the EPA also put a hold on some rules last year.”
Inventory, prioritize and segment
To start mitigating cybersecurity issues, Christensen advises operations technology (OT) staff to start with an asset inventory based on the National Institute of Standards and Technology’s newly updated NIST 2.0 framework that was validated earlier this year. It provides a sixth category on governance, which includes assessing and measuring risk, and developing suitable responses.
In general, GrayMatter advises investing in risk assessment (RA), gap analysis and risk mitigation and compliance assessment to get started. For organizations that have already done it, next steps can include incident response planning, custom solutions like network segmentation, and a strategy to augment existing network security workforces.
“There are a couple of paths we can take to remediate cybersecurity issues,” explains Christensen. “The first is getting a network segmentation plan in place, which is easier lately because segmenting can now be done without rearchitecting an entire network by using a software-defined perimeter (SDP) that determines network access like a physical switch serving as a firewall.”
Software-based security and maturity
Christensen reports that software can create deny-all philosophies, such as zero-trust, which only allow network access and communications that are prescribed ahead of time. This is similar to the older strategy of whitelisting, which only allows communications at certain sizes, speeds and times. This is different than blacklisting, which allows all communications, and then kicks out bad items that it finds. Christensen explains that whitelisting is a subset of zero-trust and deny-all strategies because it prevents all communications unless they come from known, well-behaved sources, which define good behavior and baseline it, so they know what to deny.
Christensen reports that the best of today’s cybersecurity programs are extensions of process safety efforts and standards established 20 years ago, which have been carried through in guidelines for basic cyber security hygiene, such as NIST 2.0 and standards like ISA/IEC 62443. Over the years, GrayMatter even developed its own OT Cybersecurity Maturity Model based on the collective experiences of its many clients. The model details the present security environments and capabilities that GrayMatter found among its clients, and defines their relative risks and the solutions they need to improve their cybersecurity protections and postures (Figure 1).
“The model’s range is the increasing risk vectors, which require users to address and solve the cybersecurity tasks at Level 1 before they can work on the subsequent levels,” says Christensen. “With clients, we talk about where they want to be. Is segmenting their networks, backing up their data, and having a disaster recovery plan on Levels 1 and 2 enough? They need to complete these tasks before they can do anomaly and breach detection on Level 3, which is where we say the average client should be. This is because detection requires comparing network traffic to baseline measurements that are made possible by completing Levels 1 and 2. Users must know what real assets they have and how they’re performing and communicating before they can identify anomalies or develop fake assets to serve as honeypot traps for intruders.”
Deceive and defend
In fact, GrayMatter codeveloped and launched its GrayMatter-Guard deceptive technology product three years ago, which creates decoys by using contextualized rules and filtering to meet and talk to other networks, learn what communications are allowed, and present them with deceptive assets, such as PLCs, HMIs and VFDs that represent the devices and network they’re sitting in front of. They don’t allow intruders to go any further into networks than the fake devices they’re communicating with.
“These honeypots divert and react to cyber-probes and likely intrusions, but they also show how intruders are going after our networks,” explains Christensen. “This turns intruders into our penetration testers. The beauty part is we can remediate cyber-intrusion at the same time that they’re attacking fake attack surfaces. Honeypots have been used by IT for a long time. They’re designed as educational mechanisms, which learn how intruders get into networks and behave, so users can improve their cybersecurity.”
Christiansen adds that GrayMatter-Guard lets users kick out intruders immediately, so they don’t have to wait to remove malware or risk plant-floor downtime or service outages. “Unlike IT that can take devices and systems offline and immediately apply patches, OT often can’t take systems offline to apply security updates because they may cause downtime, such as temporarily stopping customers’ water faucets,” adds Christensen. “This is where GrayMatter-Guard can help.”
Awareness and visibility for LNG
Even though cybersecurity begins with asset awareness, Christensen reports that many of the asset inventories it depends on are old, incomplete, and don’t have enough data about the behaviors of the devices in them. For example, GrayMatter recently worked with a large liquid natural gas (LNG) facility that had plenty of cybersecurity capabilities, including costly firewalls, intrusion detection, and OT visibility tools. However, within the first few minutes of GrayMatter’s visit, it found five PLCs that had been trying to dial Russian IP addresses for about six months.
Christensen reports that GrayMatter remediated this situation by increasing visibility into the LNG company’s assets and their behavior, and tuned their baseline intrusion-detection functions for greater accuracy.
“We evaluated what worked best in this situation, and for this LNG application, what proved to be the most useful was contextual filtering right below its perimeter firewall and in front of its assets, groups and facilities,” says Christensen. “We partner with Packet Viper to use its software with GrayMatter-Guard. This type of filtering detects behaviors outside of its baseline such as geofencing, and only allows traffic that originates from or is destined for the U.S. or other predefined locations. It establishes rules that authorize participants to join my network. All other traffic is blocked. In fact, we’ve learned that 60% of network traffic is usually white noise, and contextual filtering drops this out too, which also improves latency, bandwidth and flexibility.
“I think the biggest lesson for everyone is that no one is running an industrial network security program that checks all the boxes. There is always room to improve, especially with the evolving threat landscape.”
Two other examples GrayMatter often shares are:
- A customer conducting an assessment was surprised to discover that plant-floor operators could access YouTube via their HMI terminals. Exposing HMI terminals to the Internet represented a significant risk that could’ve resulted in real harm if it continued to go unnoticed.
- A manufacturer’s firewall had an astonishing 17 “any-any” rules set up in the configuration, essentially saying that any device on the industrial network could talk to any other device on the network. In the case of an attack, that attack would’ve been able to move through the network with ease.
Visualization keeps up with the future
Just as deploying basic cybersecurity measures lets users move to more advanced protections, Christensen reports that, once operating assets are visible and baseline cybersecurity hygiene is in place, many more defenses be added.
“NIST is already looking at ratifying post-quantum encryption, which uses rotating encryption keys, and reportedly runs at up to 1,000-bit bit encryption and above,” says Christensen. “Despite these potential gains, we still preach defense-in-depth cybersecurity that layers defenses, doesn’t rely on any single technology or method for protection, and uses different suppliers and products, especially between the OT and enterprise levels.”
Christensen adds that the security information and event management (SIEM) method can help visualize devices on networks, aggregate information from them, and detect, analyze and respond to cyber-threats.
“Network segmentation and zero-trust crucial, but if some parts of a network can’t be segmented, then visibility intrusion detection become even more important,” explained Christensen. “Visualization typically begins with passive monitoring and control of network traffic, usually with software from Dragos, Nozomi, Claroty, Dark Trace or others.”
Once an intrusion or cyber-attack has occurred, Christensen adds that data and operational recovery become the priority so production in the business can continue. IT departments usually possess data backups, but even though OT focuses on process up time, it could also benefit from better centralized data, archiving inversion management of its software software tools that can help with backup and recovery include Auvesy’s Octoplant, Rockwell Automation’s Asset Centre, and Copia’s various packages.
“It’s also important to know when and where changes were made to important documents and programs. Tracking versions can help if users need to go back to before an item was compromised,” adds Christensen. “The biggest malware problem for OT is ransomware, but it only works if data can’t be recovered. Consequently, the best way to counteract ransomware Is dilute what it maliciously encrypted worthless by quickly restoring data to its most recent good version. If a user can restore most or all of what’s been lost, then they don’t need to pay a ransom to get it back.”
