False process sensor data can be catastrophic but are not adequately addressed

The impetus for this blog was twofold: first, a Concordia University study dated Jan. 24, 2024, which concluded that tampering with the electric system sensors could cause grid fluctuations, and second, a meeting with the engineer who scientifically documented that radiation monitoring outside the Chernobyl nuclear plant was compromised with false data. Russia, China and Iran are all aware of the possibility of compromising process sensor data. And they’ve also all exploited such vulnerabilities.

Background

Process sensors are like the eyes, ears, and nose of any physical industrial process. These sensors measure pressure, level, flow, temperature, voltage, current, etc. The data the process sensors generate are fully trusted by default and are directly relied on for reliability, availability, productivity, situational awareness, process safety and cybersecurity.

Yet process sensors are often not included in cybersecurity programs. Operational technology (OT) network monitoring programs assume that process sensor readings are uncompromised, authenticated, and correct. This is a dangerous assumption. Unintentional and malicious false sensor data have directly contributed to catastrophic failures in nuclear plants, fossil and hydro plants, petrochemical plants, electric grids, transportation and manufacturing. Yet, OT cybersecurity regulations and guidance in general do not address the integrity of the process sensors.

Analytical studies

The Concordia study stated:

“[I]f malicious actors were able to penetrate the local area network of the converter station on the wind farm side, these actors could tamper with the system’s sensors. This tampering could lead to the replacement of actual data with false information. As a result, electrical disturbances would affect the offshore wind farm at the points of common coupling. In turn, these disturbances could trigger poorly dampened power oscillations from the offshore wind farms when all the offshore wind farms are generating their maximum output. If these cyber-induced electrical disturbances are repetitive and match the frequency of the poorly dampened power oscillations, the oscillations could be amplified. These amplified oscillations might then be transmitted through the HVDC system, potentially reaching and affecting the stability of the main power grid. While existing systems usually have redundancies built in to protect them against physical contingencies, such protection is rare against cybersecurity breaches. “The system networks can handle events like router failures or signal decays. If there is an attacker in the middle who is trying to hijack the signals, then that becomes more concerning.”

These issues aren’t confined to wind farms. Texas A&M University performed a similar study on the consequences of compromising process sensor data used in grid forming inverters for solar applications. In one case, the load current sensors were attacked to falsely show an increased demand for feeder power, which caused the breaker to trip interrupting power. In another case, the frequency and voltage sensors were compromised leading to compromise of both performance and safety.

Actual Incidents

Real-world cases in multiple industries demonstrate and support the conclusions of the Concordia and Texas A&M studies. Here are a few of them.

Electric grid: A combined cycle power plant in Florida suffered significant load oscillations because a voltage sensor provided erroneous input to the steam turbine controller. The controller reacted by cycling the turbine resulting in 200MW load swings. These oscillations from one sensor in a power plant caused a 0.25Hz impact on the entire Eastern Interconnect and resulted in a 50 MW load swing in New England (local failure affects entire interconnected systems like Colonial Pipelines).
Substations: At least some large Chinese-made power transformers installed in the U.S. power grid have hardware backdoors. As noted in Presidential Executive Order 13920, the Chinese transformer issue is a hardware, not network problem. Since there is no process sensor authentication, spoofed sensor signals “from Beijing” can take control of these Chinese transformers installed in the U.S. grid.

Manufacturing: A large industrial facility studied plant productivity. In this case, process sensors monitoring pressure, temperature, flow, motor amperage, vibration and valve position were monitored at the physics level using machine learning to compare readings to the existing Windows-based display indications. The raw, unfiltered sensor signals (the 4-20 milli-amp electrical currents) were collected on a local network not connected to the OT network and sent to a data acquisition to perform the machine learning. As the raw sensor data is “ground truth,” this approach provided an independent validation of the integrity of the OT monitoring system.        

The physics-based monitoring indicated more than half of the process sensors were either inoperable or out of calibration. Additionally, the feed pumps were having performance issues. The Windows-based SCADA did not identify these issues.           

A detailed financial analysis of the impact of the erroneous sensor readings showed a 3% impact on net productivity in this billion-dollar manufacturing facility.

Nuclear plants: During the early hours of Feb. 24, 2022, the radiation monitoring system around the Chernobyl nuclear plant exclusion zone recorded a massive increase in external gamma dose rates across 40 stations, with the most affected stations reporting dose rates three times to 600 times higher than normal. These increases vastly exceeded normal fluctuations caused by, for example, weather or surface-disturbing construction activity.

Following the anomaly, the affected stations then dropped offline. When connectivity was restored briefly between Feb. 28 and Mar. 1, 2022, radiation levels at all stations that had come back online had returned to expected values. This dose rate anomaly generated significant concern and attention in the days following the Russian invasion. Reporting by the International Atomic Energy Agency, international media outlets as well as the State Nuclear Regulatory Inspectorate of the Ukraine, indicated this temporary elevation was due to radioactive dust kicked up by invading Russian forces. If the reported dose rates were in fact valid, they would have presented an immediate and enduring hazard to all onsite.

However, based on physical constraints, data from a secondary network and statistical analysis, it can be stated with high confidence that this “radiation spike” was not due to radioactivity. The evidence instead suggests this may have been a synthetic event resulting from data manipulation.

Analysis of the location, timing and reported radiation values shows that the observed pattern could not be generated by a resuspension of radioactive material as suggested by most reporting. Likewise, this analysis strongly rules out a natural process (such as resuspension or a radiation release from a facility) having generated the values reported.

Long before Chernobyl became a byword for nuclear accident, the Three Mile Island core melt was affected by erroneous sensor data provided to the control room operators by the pressure sensors. In this case, the pressure sensors were affected by a manufacturing flaw that compromised the sensors’ ability to follow the process. As a result of the inability of the sensors to follow the process, appropriate operator actions were not taken.

Oil/Gas facilities: The Buncefield fire occurred at an oil storage facility on Dec. 11, 2005, in Hertfordshire, England. The terminal was the fifth largest oil-products storage depot in the U.K. The tank was fitted with an independent high-level switch (IHLS) set at a higher level than the Automated Tank Gauge (ATG) alarms. The IHLS failed to register the rising level of petrol, so the “final alarm” did not sound, and the automatic shutdown was not activated. As a result, the level within the tank exceeded its ultimate capacity and petrol started to spill out of vents in the tank roof. There were no fatalities, but over 40 people were injured.

The ensuing fire, the largest seen in peacetime U.K., engulfed over 20 fuel tanks. Tank 912 was fitted with a new independent high-level switch. However, the way the switch was designed, installed and maintained gave a false sense of security. Failure of the ATG system was the other immediate cause of the incident as the servo-gauge had stuck causing the level gauge to “flatline”. The “flatline” sensor impact was similar to the Bellingham, Washington Olympic Pipeline rupture, where the sensors were set to average values when the SCADA system failed. The Texas City refinery explosion that killed 15 and helped usher in process safety was affected by erroneous process sensor readings.

Hydro: The Upper Reservoir of the Taum Sauk Pumped Storage Project in Missouri was overtopped during the final minutes of the pumping cycle on the morning of Dec. 14, 2005. The precursor to this event was first identified on Oct. 3-4, 2005, when utility personnel discovered that the conduit which housed the level sensors monitoring reservoir levels was not properly secured to the dam. Deterioration of the instrumentation tie-down allowed the conduits to move adversely impacting the reservoir level readings leading to erroneously low indicated readings. The apparent low-level indications resulted in the pumps being remotely actuated overfilling the upper reservoir leading to the collapse of the reservoir.

Regulatory concerns

Those who look to regulators for help with sensor security so far look largely in vain. Process sensors are out-of-scope for the NERC CIPs and NERC Supply Chain issues. Process sensors are not addressed in EPA water/wastewater, TSA pipeline or other industries’ cybersecurity requirements/guidance. Regulations have yet to catch up with this class of vulnerability.

Conclusion

Analytical studies and actual case histories in multiple industries demonstrate that unintentional or maliciously compromised process sensor readings can cause catastrophic failures. However, process sensor integrity is generally neither addressed in cybersecurity assessments nor in cybersecurity training. When will policy makers, regulators, and cyber defenders address process sensor cybersecurity that affects every physical infrastructure?