Just like brushing and flossing, putting on pants, taking a walk or grabbing a cup of coffee, cybersecurity must become routine. It must be a familiar, everyday, even comforting activity to be effective—just as eternal vigilance is the well-known price of freedom.
Locks, firewalls or other barriers that get bolted on in a one-time panic won’t stay secure for long—no matter how much everyone wants a silver bullet or black box they can install and forget about. As always, there’s no “set it and forget it” when it comes to cybersecurity.
Think through for best fit
“To me, cybersecurity is nothing more than specialized risk of dealing with computer systems. A lot of software and devices get put in for the sake of security, but not much thought goes into why and what the organization is trying accomplish,” says Alan Raveling, operational technology architect at Interstates in Sioux Center, Iowa, a certified member on the Control System Integrators Association (CSIA). “We talk about reducing and mitigating risk to tolerable levels, but too often, organizations implement cybersecurity just to show how much they can put in, and they’re proud of it. However, when we ask how much it reduces risk or how much money it saves, it’s tough for them to quantify. Many company boards and accountants say ‘we have to do something about cybersecurity,‘ but it’s difficult later to say enough was done because no one gets in trouble for doing more. This is what leads to just adding more cybersecurity without asking what areas really need it.”
Raveling reports that some users also address cybersecurity for applications and systems using Microsoft Windows, initiate endpoint protection and antivirus software, and users’ behavior. But at the same time, they don’t address their 15-year-old controllers that are 10 revisions behind on their firmware. “Windows does have many vulnerabilities, but it still gets too much attention, while there’s not enough focus on plant-floor equipment, especially safety devices that may be linked to a network,” explains Raveling. “There’s also more potential danger because the number of network connections is increasing so fast, and that means increasing exposure and points for potential cyber-attacks."
Cybersecurity—just another risk
Raveling has worked on several cybersecurity projects at Interstates in which the challenge was getting clients to calculate and quantify the risks they faced. This was needed before they could talk about what cybersecurity measures made the most sense given their limited budgets. He adds it’s still helpful to think about cybersecurity in the same way as process safety.
“As process hazards analyses (PHA) play out in a facility, the same process and procedures can be used for security,” he says. “So, when we’re identifying where dangers exist, it could be an ammonia tank or a contractor using a virtual private network (VPN) portal. We’ve done this exercise with engineers at several clients, and it helps them learn to be more receptive to cybersecurity. Many of them aren’t cybersecurity savvy, and it can be hard to translate information technology (IT) and cybersecurity concepts that can impact their processes. Part of our job is doing this translation because we have one foot in the IT landscape and the other firmly planted in industrial operations and controls.”
However, Raveling cautions that translating between IT and OT must be a two-way street, with OT communicating its concerns back to the IT team, especially when operations has to manage software patches and reboots. “If OT is required to run some antivirus software that could interact with their plant-floor SCADA system, they must get IT to help make sure it won’t stop that SCADA system.” says Raveling. “There are many cybersecurity guides and prescriptive documents, but many users don’t know they exist. In addition, many engineers have had bad experiences with IT, so they’re hesitant to try again. Once again, Interstates sometimes mediates in these situations, and tries to get OT to give IT a second chance. OT can often review what IT is planning, and maybe tweak and change their approach where needed.”
Pre-pilot for CPG project
System integrators can further reassure the OT side by demonstrating that digitalization and other IT-based technologies won’t break their process operations, and that the risk of unplanned downtime is small and manageable. “We recently worked with a consumer packaged goods (CPG) client on the IT side of their network, whose corporate level had switched antivirus products, and wanted to do the same on their OT side. The plant-floor people were very worried about performance and compatibility issues, and how the new antivirus software would negatively impact operations,” says Raveling. “Consequently, Interstates engaged them in a small, pre-pilot project, set up an independent testing environment, and copied their HMIs and production lines. Next, we collected metrics, put the project under a regular load, and found that the new antivirus on their HMIs would affect them the same or less than the antivirus they were using before. This pre-pilot gave the OT staff enough confidence to go ahead and pilot their new Crowdstrike antivirus software on several production lines and plants. Those tests were successful, too, and data from them gave the user even more confidence to implement these antivirus protections.”
Interstates was involved in all three phases of the CPG client’s antivirus effort, including the pre-pilot, pilot and larger implementation. “The biggest hurdle to cybersecurity wasn’t the technology. It was providing the stakeholders with the assurances they needed to feel good about moving forward,” explains Raveling. “Again, this begins with identifying OT’s concerns about safety, plant stoppages, quality issues and revenue. Engineers’ reputations are typically closely tied to their production lines, and many feel like it’s a personal attack if they’re not running. Meanwhile, IT’s concerns often focus on documentation that’s driven by the chief information officer (CIO), chief information security officer (CISO), accountants and board. However, IT often takes on these mandates without enough background on why OT is concerned, and may not know why OT’s concerns even exist. Good companies know why OT’s concerns exist. Bad companies just push IT mandates forward without taking time to understand OT’s concerns.”
Learn to be on the same side
To repair these rifts and build bridges between IT and OT, Raveling reports that plant-floor people can proactively build relationships with individuals at or near the same level on the IT side of their organization. “They also need to talk to each other more often than just when they’re mad,” says Raveling. “If they want to, OT engineers can also learn cybersecurity concepts. If not, they’ll have to find someone to act on their behalf. Just like with translating, Interstates has also been asked by OT people to reframe their concerns into forms that IT can understand. System integrators can also add value by sharing their experiences with other OT clients, and give IT specific examples of bad experiences due to pushing through IT-based mandates without considering OT’s concerns. These are much more effective than hypotheticals, and will often cause IT to pause and be more sensitive to the OT side. We want them to engage with each other.”
Raveling reports that Interstates worked with a value-agriculture manufacturer that wanted to consolidate its network switches to save revenue, and allowed an IT company to take over managing all the new switches. These included switches on the plant-floor, which OT previously contracted support for, and had response agreements that matched its performance expectations.
“Now that its switches were in the IT department, the plant-floor wanted the same 30-minute response times as before, but those times went up to two hours on average. So, when there was a line outage, they were now waiting one to three hours, which meant more and longer downtimes,” says Raveling. “We’ve seen the same when IT takes over firewalls—slower responses and longer downtime impacts. When firewalls are managed by OT, their rules are more accurate and diligent. However, when firewalls are managed by IT, more mistakes occur, more job tickets are needed, and fixes take more time to complete because IT often lacks the necessary knowledge. Likewise, when new devices are installed, it typically takes one to three extra days if less-expensive, IT-based switches and firewalls are implemented. Without the skills, experience and knowledge of OT’s rules for firewalls, they just take longer to implement.”
Despite these drawbacks, Raveling concludes it’s still important for OT to build relationships with IT, so they can learn each other’s concepts and priorities. “The challenge for IT is learn more about the OT languages and ways of thinking, and understand which are their most valuable and risky assets,” adds Raveling. “This will let them start to talk about what to protect with cybersecurity, and get input from both sides.”
