This blog started out as a response to Dale Peterson’s May 11, 2022 S4 interview with Ilan Gendelman of Siga OT Solutions, but I have extended its scope to all infrastructures because if you can’t trust what you measure, you have no cyber security, process safety, or product quality. It is unfortunate that process sensor issues are not being addressed by the U.S. Department of Energy (DOE) the U.S. Department of Homeland Security’s (DHS) Cybersecurity and Infrastructure Security Agency (CISA), the North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection (CIP) standards for the electric industry, the American Water Works Association (AWWA) cyber security guidelines, the Transportation Security Administration (TSA) pipeline cyber security requirements, the American Petroleum Institute (API) cyber security guidelines, the American Gas Association (AGA) cyber security guidelines, among others. This is why my March 10, 2022 seminar to the U.S. Air Force Cyber College was entitled “Shields Up and Good Cyber Hygiene Do Not Apply to Insecure Process Sensors”.
S4 Interview
As mentioned, May 11, 2022 Dale Peterson had an interview with Ilan Gendelman of Siga OT Solutions on his S4 YouTube channel. Because my name was liberally “thrown around”, I had a follow-up conversation with Dale on May 13th. Dale and I are coming from different directions – I focus on process and process sensor integrity for safety, reliability, and cyber security (engineering). Dale’s focus is on the networks and more traditional cyber security. Consequently, Dale had questions about sensor monitoring including how many sensors need to be monitored and how important is the fidelity of the sensor signals for cyber security. Hopefully, this blog can answer Dale’s questions.
For those unaware, I supported SIGA OT through April 2019 and so have background on their process sensor (Purdue Reference Model Level 0,1 devices) monitoring technology. There are several other companies that purport to do Level 0,1 monitoring. As my only knowledge is from those vendors’ websites, I am not in a position to say anything about their actual capabilities.
Process sensors can affect cyber security
Every infrastructure, whether building controls, power plants, electric grids, water/wastewater systems, pipelines, manufacturing, transportation, defense, etc. start with measuring pressure, level, flow, temperature, voltage, current, etc. The process sensor measurements are used for both engineering applications to monitor and control physical processes in real time, which may not have direct cyber security implications. The process measurements are also input to Operational Technology (OT) networks which have cyber security implications as these measurements have neither security nor authentication.
Process sensors have direct or indirect connectivity to the Internet with no cyber security capabilities (see https://www.controlglobal.com/blogs/unfettered/a-vulnerability-worse-than-log4j-and-it-can-blow-up-facilities-and-shut-down-the-grid/). Process sensors, whether analog or digital, need to be periodically calibrated to assure accurate readings. The calibration devices or calibration software have no cyber security and yet are connected to the Internet. Two examples of this vulnerability are porting the calibrator software for the handheld process sensor calibrators to cellphones and the addition of Bluetooth capabilities. The first vulnerability case is the cellphone Highway Addressable Remote Transducer (HART) Calibrator. The advertisement states that: “a key advantage of a mobile app solution over traditional handheld HART communicator is that you can use the mobile device you already own. In addition to already owning the main piece of hardware required, it is typically upgraded every couple of years for a very low cost (if not for free). You are continuously getting more features and more processing power without any effort.” The second advertisement addresses the use of Bluetooth: “In the past it was difficult to use a handheld or even a PC-based HART communicator in hard-to-reach places. Carrying a laptop up a ladder is dangerous. However, now there are Bluetooth-based HART modems that provide great convenience. Connect the Bluetooth HART modem in the hard-to-reach location, and then climb back down to safety and use your mobile app-based HART communicator safely from the ground.”
Because of this Internet connectivity, all process sensors that are on the Human Machine Interface (HMI) network (generally Microsoft Windows-based operator displays) as well as in process Historians, can be altered or have malware installed before any network security protections can be employed. This cyber vulnerability is independent of the significance or insignificance of the process sensor measurement to the overall process.
The Instrument Asset Management Systems (IAMS) is where process sensor data reside. IAMSs have no cyber security and use cyber vulnerable protocols. Moreover, the IAMSs have direct connections to the Internet, Enterprise Resource Planning programs (e.g., SAP, Oracle, etc.), and the process sensor settings. In 2016, the Russians demonstrated they could hack the process sensors via the cyber vulnerabilities in the IAMS. As mentioned, these cyber vulnerabilities also can be used to access the ERP or the process sensor settings.
Process sensor fidelity
Important information about the health of the physical processes and the process sensors are found in the milli-second to second “squiggles” in the process sensor readings. Consequently, engineering requires high fidelity (higher frequency) monitoring of the process sensor signals to analyze the “squiggles”. High fidelity monitoring of the higher frequency content is the mechanical analog of intrusion detection and is used to monitor equipment issues such as equipment or flow-induced vibration, instrument tube plugging, instrument jitter, and other process and sensor issues. The concern is the higher frequency content cannot be found in Internet Protocol (IP) networks because of the inherent time delays (seconds to minutes) in Windows and other general purpose operating systems. This means that this “squiggle” data is effectively filtered out. In one instance, a bad sensor reading in a utility-scale combustion turbine (not off-scale but outside the operating limits) prevented the combustion turbine from restarting. The HMI readings, without the higher fidelity, did not indicate a problem. The root cause (“bad” temperature sensor) was found from the raw, unfiltered sensor data. Because the higher frequency information is not directly relevant to cyber security, network cyber security personnel were unaware of the consequence of this engineering data not being available. As Jason Larson’s S4 triangle presentation and the Stuxnet sensor attack demonstrated (https://www.controlglobal.com/blogs/unfettered/you-cant-protect-the-unprotectable-our-critical-infrastructures/), sensor data can be maliciously exploited and not be detected from network monitoring. That is, the attackers are aware of process sensor cyber limitations, the defenders refuse to address it – a bad situation to be sure.
How many sensors should be monitored
A large power plant or refinery may have 20,000 to 40,000 process sensors. Consequently, Dale asked Ilan how many process sensors need to be monitored in a facility. Ilan’s answer was that 7-10% can provide a representative sample. However, the number to be monitored depends on why the sensors are being monitored. One concern is the proliferation of counterfeit sensors. Taking a small representative sample would not address the concern that all potential counterfeit sensors need to be identified. Another concern is from a field test that occurred in the late February 2022 time frame at a manufacturing facility. In this case, the test was being done for product quality and reliability concerns not cyber security. Consequently, all process sensor raw data on the manufacturing line were being monitored by the process sensor monitoring program. The facility had no reason to suspect there were any concerns with the line as material was going in and product was coming out and the HMI appeared to be working properly as filtered sensor data was being displayed. In this case, it was not just the sensor monitoring, but the engineering analysis of the monitoring data by JDS Energy and Mining (JDS) that identified the importance of the process sensor issues. JDS found that more than half of the sensors were either inoperable or out-of-specification during some of the identified events. However, the HMI gave no indications there were any faulty sensors. The sensor issue prompted the instrumentation department to review non-monitored sensors. This additional review identified calibration issues with at least two unmonitored plant sensors. None of the monitored and non-monitored sensors were considered faulty prior to the investigation. The number of “failed” sensors was a significant product quality issue as well as a cyber security issue as the failures could either be from unintentional or malicious causes. As a result, the facility wanted to look at more processes to determine if there are other similar process sensor issues. It is evident that monitoring only 7-10% of the sensors and not having the engineering expertise to interpret the results is not acceptable. For productivity, product quality, cyber security, and process safety, it was important to identify the sensors that were providing questionable data.
Some general observations about the industrial Internet-of-Things (IIoT)
It’s important to remember that sensors, in modern systems, are almost always connected devices, part of the Internet-of-Things (IoT). They can be affected directly, or their connection to control networks can be meddled with. Consider a report May 16, 2022 from NCC Group, whose researchers demonstrated that Bluetooth Low Energy (BLE) systems are vulnerable to link layer relay attack. The news was generally reported with headlines that talked about how car thieves could in principle steal your Tesla, but the problem is more widespread than that. BLE is, NCC Group explains, "the standard protocol used for sharing data between devices that has been adopted by companies for proximity authentication to unlock millions of vehicles, residential smart locks, commercial building access control systems, smartphones, smart watches, laptops and more. What makes this powerful is not only that we can convince a Bluetooth device that we are near it—even from hundreds of miles away—but that we can do it even when the vendor has taken defensive mitigations like encryption and latency bounding to theoretically protect these communications from attackers at a distance." Now consider that BLE can also be used for calibrating process sensors as mentioned previously.
Summary and recommendations
While no one would argue that network security isn’t important, it’s also important that the basic process sensor data that cross the OT network not be overlooked. Process sensors are necessary input for reliability, availability, safety, predictive maintenance, product quality, and cyber security. Yet process sensors have no cyber security and are connected to the Internet during maintenance potentially introducing malware or sensor manipulation. Attackers are aware of process sensor cyber limitations as demonstrated by Stuxnet, yet the defenders refuse to address it – a bad situation to be sure. Monitoring and analyzing the raw sensor data identified previously unknown sensor failures. Because of the Internet connectivity and unknown sensor failures, all process sensors in critical processes should be monitored even if they are only on local networks. Additionally, because of their Internet connectivity, all process sensors that are on the Windows IP network need to be monitored even if they are not considered critical. Domain engineering expertise is needed to understand the implications of process sensor monitoring.
Joe Weiss
