Because it’s an engineering company that also provides cybersecurity capabilities, Cybertrol Engineering in Minneapolis, Minn., typically starts its projects by asking each customer what they know about their own systems.
“They may know plenty about what they’ve got they or they may know nothing, but the discovery process creates a picture that identifies what’s there, and shows end-of-life components and what should likely be done to go forward. It also provides advice on network topologies and infrastructure changes that may be needed, such as adding servers or other upgrades,” says Alexander Canfield, industrial information technology (IIT) manager at Cybertrol, which is a certified member of the Control System Integrators Association (CSIA) that works mainly in the food and beverage, life sciences, chemical, and medical device industries. “With cybersecurity, there’s a lot more discussion and analysis lately about who should do it, information technology (IT) or operational technology (OT), what their roles should be, who owns items like passwords, where they should be kept, who can access them and other related issues.”
Get your subscription to Control's tri-weekly newsletter.
To sort out these issues, Cybertrol conducts cyber-risk assessments (cyber-RA) along with its discovery process, but it also employs Cisco’s Cyber Vision software, which listens passively to networks where it’s deployed, and builds a functional map of which devices are talking to each other and what they’re talking about. Other passive-monitoring software includes packages from Verve and Fortinet. “Cyber Vision is especially helpful for OT because it doesn’t ruffle or interfere with plant-floor devices that often can’t handle typical network-sniffing software,” says Canfield. “It also analyzes firmware, other components and software for vulnerabilities, and gives grades and advice about how to harden them. Its more advanced version can also do constraint monitoring of processes, but we’re presently only using the local version on our laptops.”
Canfield reports the difficult cybersecurity challenge of seeking out and investigating what’s not known is compounded by the growing pains of IT and OT convergence and the ongoing turmoil of digitalization.
“IT often doesn’t understand what OT needs to know, the issues and causes it’s trying to discover, or that regular patching or network evaluation tools may disable OT equipment,” explains Canfield. “For example, the recent CrowdStrike problems were related to deploying Day 1 updates. However, OT must test, and prove that patches and other updates are solid before deploying them. We use Rockwell Automation’s Microsoft Patch Qualification website that tests Windows updates against its software. We also do this as part of individual support contracts that test and apply patches during dedicated support visits.
Cybertrol also follows Cisco’s recently updated Converged Plantwide Ethernet (CPwE) cybersecurity guidelines, which call for using managed Ethernet switches as firewalls to segment networks, especially between plant and business sectors. CPwE also recommends performing multifactor authentication (MFA) with software, such as Cisco’s Duo two-factor authentication software.
Zero-trust and proxies
To further limit questionable network traffic, Cybertrol uses newer zero-trust strategies, which are similar to older whitelisting procedures that only allow communication between predefined devices. However, zero-trust is different and more advanced because it directs network traffic to permitted destinations by learning and baselining what’s allowed to talk to what, prompting users that acceptable communications are occurring, and allowing traffic to move back and forth.
Similarly, Cybertrol employs software proxies, which are look-up tables that direct traffic from internal devices to reach specific services based on lists of actual destinations and directions. Authorized communications that ask for an authorized destination are sent to it. However, if a device doesn’t ask for the right direction, then proxies provide defense by preventing its communications from going anywhere. Software aided by artificial intelligence (AI) is expected to make the listening, learning and list-building capabilities of proxies and zero trust easier and more effective.
Canfield adds that proxies can be deployed by installing a local software service on a virtual machine (VM) running Linux or another operating system, which handles communication handshakes, and funnels authorized traffic through the demilitarized zone (DMZ) between the firewalls that separate network segments. “The choice of a local service and VM usually depends on which SCADA/HMI platform the user is running,” explains Canfield. “Operating systems like Linux can do this, but others such as Windows can do it just as well. These services can also run in a software containers like Docker, but that technology is relatively new, so it’s not as widely used yet. We can also put them in an onsite VM by using the functional design map and running services in the DMZ.”
Solving switch issues
Beyond directing communications and resolving patch issues, Cybertrol also helps users solve individual cybersecurity problems. For example, one of its food and beverage clients recently experienced a network outage just after adding managed switches to Layer 3 of the existing network at its brownfield facility. These stacked, out-of-the-box switches were deployed as a main distribution frame (MDF) because the customer’s IT staff was working with their OT colleagues, and they wanted to remotely access the OT network.
Unfortunately, the new switches were connected without being configured, or even being assigned Internet protocol (IP) addresses, and this caused the whole network to crash when the users tried to add firewalls to the MDF. This occurred because these networks typically use spanning-tree protocols to prevent loop problems, but this requires that all devices use the same types of address assignments and configurations for the network to function, otherwise they’ll conflict with one another. In addition, the food and beverage manufacturer also had mismatches in its virtual local area network (VLAN), and this created conflicts between the new switches and the IT network.
“When we set up spanning-tree protocols within networks, we define one switch that’s in charge, otherwise the network tries to figure it out by going through its media access control (MAC) addresses,” explains Canfield. “In this case, we also went to the network and added basic Layer 3 programming. This lets the devices keep functioning until an overall network analysis could be conducted, where IP addresses, names, spanning-tree topology and VLANs could be mapped out for further remediation. These are steps to take when you’re looking to increase cyber-resilience.”
Another difficulty was the food and beverage manufacturer had multiple plants that were each running their own types of OT networks with various stages of cybersecurity measures in place. To resolve these differences, Cybertrol analyzed each facility, and collaborated with their corporate IT department to implement companywide standards for OT networking and cybersecurity. “This was possible because we’re not just an engineering company. We have dedicated OT specialists, who can also talk IT and vice versa,” adds Canfield. “Most companies already have IT personnel. However, OT and process engineering creates the controls, which means companies need engineering and controls guys, who also know IT, so they can mediate between them.”
