Well begun truly is half done, but with cybersecurity, it's still hard to progress beyond initial steps like updating passwords and segmenting networks with managed Ethernet switches used as firewalls to monitor network traffic, detecting suspicious behavior, and responding to threats and attacks.
"We started our cybersecurity journey at the tail end of a smart grid deployment when we realized we had a large amount of data that wasn't going to be useful if we couldn't operationalize it, which meant we had to get comfy with our security group," says Jason Nations, senior enterprise security manager at OGE Energy Corp., Oklahoma City, who spoke at ARC Industry Forum 2019 earlier this year in Orlando. OGE is a vertical electric utility that uses natural gas, coal, wind and solar sources to generate power for 700,000 customers in Oklahoma and western Arkansas. "We inventoried all our assets and connection, and used a line-item diagram, so we knew every cable and what it was connected to, including showing what sensors were needed. However, we also learned cybersecurity is a people problem as much as it's a technical one, and that we needed to get everyone onboard, especially to get our use cases in line. Without the involvement of our field personnel, execution was also at risk."
Nations reports that OGE aimed to implement continuous monitoring in its control system networks; gain enterprise visibility across its OT environment; perform real-time inventory of its control system cyber assets and further mature its detection capabilities; and improve its incidence response capabilities. Cybersecurity solutions had to meet its use cases; form a long-term relationship with OGE; support its commitment and capabilities; and integrate with the utility's ICS security program. It also followed the National Institute of Standards and Technology's Cybersecurity Framework and the U.S. Dept. of Energy's Cybersecurity Capability Maturity Model (C2M2); planned and coordinated deployments with field personnel; implemented undisclosed ICS threat intelligence software and components within three months; and integrates alerts from the U.S. Dept. of Homeland Security's (DHS) Industrial Control System-Cyber Emergency Response Team. Nations adds that OGE is also using John Kindervag's Zero-Trust Network model that takes a guilty-until-proven-innocent approach.
"We had to understand our operating environments better because if we didn't know our processes and they weren't visible, then we couldn't detect anomalous behaviors and limit the dwell time of threats in our system," says Nations. "Finally, we also looked for partners and vendors that could help us define our use cases. We didn't need a cybersecurity Swiss Army knife with 87 blades, if we just needed the four right ones. Many users think that cybersecurity isn't doable, and it is hard, but it can be done." But, as is always the case with cybersecurity, the story doesn't end there.
Big threat-response picture
In general, more process industry managers and corporate boards are finally realizing they must protect plant-floor operations as well as their enterprises and finances, and invest in cybersecurity, but it appears many of their efforts don't yet go far enough to monitor, detect, prevent, mitigate and recover from today's always-evolving probes, threats, intrusions and attacks.
"Users typically take the initial steps of trying to prevent cybersecurity threats by enabling anti-malware software and using firewalls to reduce the likelihood threats becoming reality. However, fewer do monitoring and detection using security information and event management (SIEM) or anomaly and breach detection, and even fewer are using threat intelligence methods to respond to cyber threats and manage their vulnerabilities," says Sid Snitkin, VP of cybersecurity services at ARC Advisory Group. "More people are needed to look at SIEM data, do anomaly and breach detection, and perform threat management."
Added cybersecurity staffing may follow technology increases because industrial and operations technology (OT) cybersecurity markets in Europe, Middle East, North America and Asia are increasing at a 14.3% compound annual growth rate (CAGR) from just over $3 billion in 2017 to about $4 billion this year, and are projected to reach just over $6 billion in 2022, according to Snitkin, who reported on ARC's research, while introducing the cybersecurity presentations at the forum.
"There are both resource and technology gaps to close on the way to industrial/OT cybersecurity maturity, but lack of resources undermines everything else. Users must triage the security they need most, invest in tools and hiring, and seek outside assistance," adds Snitkin. "OT and IT are merging, and OT needs more IT expertise and people, while IT needs more plant expertise and data. No one has enough cybersecurity people, so as OT and IT converge, their operations teams need to merge into integrated IT-OT security operations centers (SOC). Their objectives are: shared responsibility for end-to-end cybersecurity; global, corporate governance of all cybersecurity policies, procedures and technology guidelines; and continuous, global visibility and management of all cyber assets, vulnerabilities and threats. Their challenges are cultural and technical issues, process constraints, and different views of organization and management."
Watch the network
Once passwords are enabled and networks are segmented, the most pervasive cybersecurity procedure is network monitoring, traffic analysis, and threat detection and response. For example, City of Raleigh, N.C., recently deployed Indegy's Industrial Cybersecurity Suite in just a few days to its public utilities that provide water/wastewater and other services to 500,000 customers.
"One thing that was a real plus was the ability to query PLCs and get back information about what programming changes had been made to them, versioning information as changes were made that we didn't have in the past. Now, we have a time stamp on when changes were made and can determine who made them," says Steve Worley, SCADA security manager, City of Raleigh NC Municipal Government. "Other solutions do more passive monitoring, but Indegy does both the passive and active component that was a real value to us. We wanted to provide some accountability to our system integrator, who was making changes on a regular basis. We also needed to do some automated asset discovery, but it's a huge job to keep track of them, and so automation is key."
Worley adds that within minutes of installing and engaging its Indegy device, the utility was able to provide a huge amount of data on the network that would have taken weeks to collect by hand. "The automation provided asset names, IP and MAC addresses, and other things that were useful to us for network management, and didn't have to be gathered manually," says Worley. "We had it all on one screen on the Indegy console. Because some of the asset management is automated, I can spend less time on that and more time on looking at vulnerabilities and remediating them."
Read more: OGE cybersecurity checklist
Likewise, to remotely monitor its clients' panels and other equipment, Gettle Inc. in Emigsville, Pa., recently implemented mGuard Ethernet switches and mGuard cloud-computing service from Phoenix Contact to securely troubleshoot their devices and data over the Internet. "With mGuard secure cloud, we can be online with a customer in 10 minutes, and resolve their problem fairly quickly, which decreases downtime and increases productivity," says Roman Bair, lead automation systems integrator at Gettle.
The IIoT paradox
Most recently, the Industrial Internet of Things (IIoT) and cloud-computing services are trying to help with automated network monitoring, alerts, updates and even some cybersecurity policy enforcement. However, because pretty much all of these IIoT and other connections are Ethernet-based, they're an ironic gift-and-curse because they open an avenue for the very vulnerabilities and threats they must subsequently seek to prevent.
"It's difficult to know what's connected and how to harden it for greater security because we're dealing with 250 sites with production, pipelines and shipping assets, so we started with cybersecurity monitoring," says Mike Hoffman, principal ICS security engineer at Shell, who also spoke at the ARC conference. "We spent the past year and a half exploring most of the major cybersecurity solutions and vendors, and learned they perform well for their use cases, but that may not be good enough for individual users. Asset inventories are also automating, and need to maintain connectivity, but that means checking if it's OK for that OPC UA server to talk to this historian or device?"
Likewise, because Ethernet is the primary network medium, but has connections that can fail over time, Hoffman reports that users with devices that make repeated transmissions also need tools that can show their communication status and act on any changes, including possible security issues. "This is how cybersecurity becomes part of the overall uptime perspective," says Hoffman. "Keeping plants running is also achieved by making them secure. However, understanding cybersecurity also means aligning it to your particular business case, and are you dealing with an IT-based intelligence feed or an OT feed? However, when we evaluate cybersecurity solutions, we have to determine how well its software scales for us, or do we need updates to be pushed up or trickle down in a certain way."
Never-ending list
Just like the job jar for homeowners, the challenges and tasks required for successful cybersecurity never run out.
"Threat intelligence means different things to different people, so users must adopt the one that addresses the needs of their operations and environment, and focus on what will add the most value," explains OGE's Nations. "It's also important to move quickly because threat actors are aware that users will block their IP addresses."
In the near future, OGE also plans to continue vendor training; refine its inventory; begin examining tactics/tools, techniques and procedures (TTP) versus indicators of compromise (IOC); develop more alerting capabilities; refine its runbooks compilations of procedures; and conduct ICS threat hunting. IOC is forensic data that may indicate malicious activity, while TTP are practices typically used by intruders.
"We also got a lot of false positives in the beginning, so if there's a threat feed, we apply IT capabilities to it because we can't just blast out cybersecurity incident reports, which is like trying to drink from a fire hose," adds Nations. "In fact, we're trying to reduce the windows and enrich the data feeds that our analysts and other cybersecurity staffers have to look at, so they can be more effective. We still use SIEM, but getting alerts every five minutes is too much, and means we can't watch what's valuable and what's not. Services like Amazon Web Services (AWS) or Microsoft Azure can help make sense of threat intelligence feeds and other large amounts of data coming in.
"Good cybersecurity also means employing blocking-and-tackling like using NIST and DoE's recommendations before doing anomaly detection, but in the end, our cybersecurity program really took off when our products people got involved with our plant managers, whose authority is inviolate, and convinced them to partner on cybersecurity by showing what was in it for them. If you can get everyone pulling on the same rope like this, it's very helpful because then you'll have use cases that you can operationalize. We developed enough of a security culture that our field people call our security people to come out and assist them."
