Introduction
Control system cyber security is about keeping lights on, water flowing, etc. It is not simply matter of maintaining network availability. If control systems are affected by a cyber incident, whether it’s an unintentional incident or a deliberate attack, critical infrastructure reliability, availability, and safety may be impacted. Industrial, manufacturing, transportation, and others rely on Operational Technology (OT) Internet protocol-based (IP) networks to bring significant productivity improvements. However, along with those improvements come significant cyber vulnerabilities.
The fallacy about critical infrastructure cyber security is the assumption that IP networks are needed to keep lights on, water flowing, etc. For more than 80 years, the grid operated without an IP network. Control systems in power systems are designed to work in coordination with each other so the equipment associated with control systems can work without SCADA and the SCADA network. As an example, following the 2015 cyberattack of the Ukrainian power grid, the Ukrainians continued to operate the grid manually for months without the IP networks as the IP networks couldn’t be trusted. However, the grid could not be operated if the critical hardware were compromised or damaged. This includes the process sensors monitoring and controlling the grid.
July 28, 2021, an announcement was made about President Biden’s Industrial Control System Cybersecurity (ICS) Initiative which is a voluntary, collaborative effort between the federal government and the critical infrastructure community to facilitate the deployment of technology and systems that provide threat visibility, indicators, detections, and warnings. To date, this is a network-based approach specific to cyber threats. On the other hand, control system field devices such as pressure, level, flow, temperature, and voltage sensors (often not considered part of OT) are inherently insecure and generally not designed to be connected to IP networks. The President’s ICS Initiative is not addressing this problem.
Background
Prior to the 9/11 event, cyber security was simply one of the risks that had to be considered when designing and implementing control systems along with seismic risk, environmental risk, fire risk, reliability risk, etc. Those risks were regarded as engineering considerations and managing them was considered an engineering function. The intent was to ensure that the engineering basis of the design would be met, regardless of the risk. Consequently, the engineering organization was responsible, and this included cyber security. It was a “bottom up” approach of process anomaly detection, performed in the interest of mission assurance. In fact, this was the basis of the Electric Power Research Institute (EPRI’s) control system cyber security program I helped start in 2000 which unfortunately is now, like others, about securing the networks.
Sometime after 9/11, cyber security became national security. However, around the same time, cyber security for control systems was moved to the IT (now OT) network monitoring organizations with engineering no longer involved. As a result, control system cyber security went from Mission Assurance to Information Assurance. The focus on networks rather than on the process can also be seen by having the CISO and not the Vice President of Engineering/Operations responsible for the cyber security of engineering systems. Consequently, cyber security monitoring and mitigation tended to move to the IP network layer – network anomaly detection tended to replace process anomaly detection. Control system devices such as protective relays work on instructions entered into registers within the hardware of the device. These instructions reference other instructions and raw process sensor input data to perform desired commands. This means that devices such as protective relays have little to do with traditional higher-level networks but depend on the integrity of the measurement.
The reticence of the US government and industry to move away from a traditional network-based approach can be seen from the following examples:
- The July 2021 Version 2.0 of the Cybersecurity Capability Maturity Model (C2M2 does not address the process sensors and process anomaly detection. How mature can the process be if it doesn’t address what keeps the lights on and water flowing?
- The electric industry’s NERC Critical Infrastructure Protection (CIP) cyber security standards consider process sensors out-of-scope for cyber security considerations.
- The recent podcast by Idaho National Laboratory personnel supports the network approach ((https://www.synack.com/were-in-synack-podcast/?utm_source=organic_social.
- Presidential Executive Order (EO)13920 was issued following discovery of hardware backdoors in large Chinese-made electric transformers. As can be seen from the EO, it was focused exclusively on hardware and control systems. However, the government and industry response was to turn this hardware attack into a software supply chain problem.
Use of sensor monitoring
Process sensor monitoring has been used for many years for process anomaly detection. I was using it in the late 1970’s to identify flow-induced vibration issues in nuclear plants and in the early 1990s while managing the EPRI Nuclear Instrumentation and Diagnostic Program to detect a major supply chain common-cause process sensor problem.
Legacy engineering field devices such process sensors, actuators, drives, positioners, and analyzers have no cyber security, authentication, or cyber logging nor can they be easily upgraded for cyber security. Yet, process sensors deliver the inputs to the OT network where the OT network monitoring providers ASSUME the sensor input is uncompromised, authenticated, and correct. However, because the sensor input is not authenticated, it is not clear that the apparent sensor data is actually coming from the sensors and not from “spoofed” signals. The actuators, drives, controllers, etc. receiving the sensor signals have no way to authenticate the origin of the sensor signals and therefore automatically accept the sensor and respond accordingly. This could be the approach the Chinese are using with the hardware backdoors in the large electric transformer to take control of the transformer without having to hack the networks. Therefore, there is a need to take an intractable network monitoring approach and make it a tractable engineering program.
Modern machine learning enables pattern detection of the raw process sensor signals that wasn’t previously possible. It is this additional capability that enables sensor monitoring to identify process anomalies regardless of cause and independent of IP networks and their associated cyber vulnerabilities. As a result, the Israel Water Authority recently took that engineering approach, approving off-line process sensor monitoring technology to secure the country’s water systems. Unlike the prevalent US practice of monitoring IT and OT networks for cyber security (that is, for network anomaly detection), the Israeli approach is based on monitoring the electrical characteristics of the process sensors (process anomaly detection) and not just relying on network monitoring like the US.
Benefits of off-line sensor monitoring
An analogy of why the process sensor approach can be so valuable is to consider a car moving at 70 miles/hour when one of the tires has a flat. You pull off the road and replace the flat tire with the small run-flat spare. You can then continue to drive the car, albeit at a reduced speed, until you can get the regular tire replaced. Now consider ransomware. The IT and OT networks provide productivity. However, if they are lost because of ransomware or any other type of malware, the off-line monitoring of the sensors which is not sensitive to the IT malware, allows the facility to continue operating, albeit in reduced efficiency, until the IP networks are restored.
Specifically, the benefits of the Israeli approach include:
- Raw process sensor signals provide ground truth about the physical operation of the system.
- The process sensor monitoring system is not susceptible to IT or OT unintentional network issues, or network attacks (including ransomware) or vulnerabilities induced by patch management oversights.
- As process anomaly detection, the system detects any anomaly regardless of cause, not just malicious cyberattacks, which means even sophisticated attacks that look like equipment malfunctions will be identified (e.g., Stuxnet).
- I have amassed a database of almost 12 million control system cyber incidents that have resulted in more than 1,500 deaths and more than $90 Billion (US) in direct damage. However, the vast majority were not identified as being cyber-related by network-monitoring. Process sensor monitoring would have been able to identify many of these incidents as process anomalies.
- By monitoring in real-time, the system is essentially a sensor health monitoring system and so also functions as a predictive maintenance system that can be used to extend maintenance intervals.
- Process sensor monitoring systems have detected equipment impacts that were not identified by the Windows-based OT monitoring system.
- Monitoring the sensors requires the involvement of the engineers responsible for the process.
- Monitoring the process sensors provides authentication, which otherwise would not exist.
- The process sensor monitoring system is applicable to any critical infrastructure and has been installed in water, power, chemicals, and building controls.
- Monitoring of process sensors applies to all infrastructures as they all use process sensing. This approach of addressing multiple industries meets the intent of the President’s ICS Initiative. For example, the new TSA cyber security requirements do not address potential pipe failures because they are network-based and don’t address the process sensors. As of July 27, 2021, critical pipeline operators have reported more than 220 cybersecurity incidents since the US Transportation Security Administration (TSA) implemented emergency measures in the wake of the crippling ransomware attack on one of America's most important pipelines. However, I am not aware of any recent reports of pipeline ruptures or pipeline outages meaning the 220 cyber security incidents were IT incidents not affecting the operation of the pipelines. Yet the two cyber-related pipeline ruptures that have killed people and destroyed structures would not have been addressed by the TSA cyber security guidelines.
- Sensor monitoring can be applied to certain supply chain situations like the hardware backdoors in the Chinese-made electric transformers to ensure the sensing input going to the transformer devices are not “spoofed” signals coming from elsewhere as the hardware backdoors bypassed all cyber security protections.
- Given the recent JBS meat processing plant shutdowns, the sensor monitoring approach can help the food industry justify continued operation as there continues to be a view of the plant processes.
The limits of network security
The disadvantages of the US approach include:
- Neither IT nor OT networks provide ground truth about the process and assume the sensor input is uncompromised, authenticated, and correct.
- Network monitoring is a never-ending “whack-a-mole” issue (e.g., defenders come up with a solution, attackers come up with a bypass).
- Even the best network cyber security can be defeated (see SolarWinds).
- OT networks are susceptible to unsophisticated as well as sophisticated network vulnerabilities.
- OT cyber security organizations tend to exclude the engineers responsible for the design and operation of the control systems.
Summary
With the never-ending, and too often successful, attacks on critical infrastructure networks, there needs to be a better way to protect control systems and the processes they monitor and control. July 28, 2021, an announcement was made about the President’s Industrial Control System Cybersecurity (ICS) Initiative to facilitate the deployment of technology and systems that provide threat visibility, indicators, detections, and warnings. To date, this is a network-based approach specific to cyber threats. However, the existing approach of securing critical infrastructures by securing the networks is not working. The Israel Water Authority recognized that need and is monitoring the electrical characteristics of the process sensors as the raw process sensor signals are ground truth and not susceptible to network attacks. Hopefully, the US government, insurance companies, credit rating agencies, and others recognize what is really needed to be secured – the field control system equipment that keeps lights on and water flowing.
Joe Weiss
