Process sensors are mechanical/electrical devices that can fail for various reasons. “Hard” failures of the sensor occur when the sensors are no longer within the design operating range or no longer operating (e.g., fail high, fail low, fail as-is) – obvious failures. “Sensor drift” is the phenomena of process sensors deviating over time from their “absolute” calibrated value because of mechanical or electrical reasons so they are no longer as accurate as thought (what I call “subtle” failures). These subtle failures have caused catastrophic failures and also have prevented processes from operating in optimal conditions because of the loss of process sensor accuracy.
Sensor drift issues are not hard failures as the sensor values are still within the design range and the deviation is generally very small over time. However, depending on the process this can still be a reliability or safety problem. Consequently, for nuclear regulatory and plant operational reasons, I was performing detailed analyses of process sensors and equipment forensics long before I got involved with cyber security. Because I was skeptical of the long term accuracy of the sensors, one of the projects I performed while managing the EPRI Fossil Plant Instrumentation and Control Systems Program was to evaluate process sensors in actual (fossil) power plant operation for sensor drift – and yes they did! The only way to monitor for sensor drift is monitor the raw sensor signals in real time which is what we did before Ethernet networks came along. However, because of the filtering of the serial-to-Ethernet convertors, it is not possible to accurately monitor sensor drift at the Ethernet packet level as it MUST be done at the raw signal level. This can easily be overlooked if one is too quick to apply an IT template to what amounts to an engineering safety problem.
January 3rd, 2019, SecurityWeek had the following article - https://www.securityweek.com/ics-security-experts-share-interesting-stories . The following was the only case that actually affected control systems and not just identified unexploited vulnerabilities. According to Paul Smith, director, product research and strategy, Nozomi Networks: “The main priority for operators in the midstream oil and gas industry is to keep their product flowing through pipelines in a secure and safe manner. It is also critical that operators have visibility to mitigate any cybersecurity issues and detect any potential outages which could have an impact on their services. When visibility into what is really happening is reduced, significant problems and costs can arise. This is unfortunately what happened with a major pipeline organization when a PLC went down and caused the company $1.9 million in lost revenue and downtime.In this case, the problem with the PLC was that it suffered from ‘ghost drift’. This is when a device slowly and quietly slips out of scope over such a long period time that no one ever notices. When the PLC started to fail, it skewed the numbers ever so slightly that it was not noticeable, and it was only once the real damage was done that the problem was finally spotted. This scenario highlights the importance of operators of critical infrastructure deploying solutions which have the capability to detect when devices start to drift or operate incorrectly. Today’s ICS network monitoring solutions alert the operator that it’s time to take a closer look before another unplanned downtime incident wracks up a $1.9 million loss. These solutions help oil and gas operators around the world see and secure their critical industrial control networks and provide real-time visibility to manage cyber risk and improve resilience within these environments.”
There are several concerns with Paul’s statements. Firstly, there is no so such thing as “ghost drift” of a PLC as digital devices such as PLCs don’t drift. However, process sensors can drift. His statement that “Today’s ICS network monitoring solutions alert the operator that it’s time to take a closer look before another unplanned downtime incident wracks up a $1.9 million loss” is correct, but only to a point. Such monitoring would have to be at the raw sensor level not at the network level. There have been sensor hacking demonstrations that show that neither the PLC nor HMI would be aware of a malicious compromise or unintentional drifting sensor if the hack or drift occurred before the sensor input became an Ethernet packet. Sensor drift is apparently what occurred in the Nozomi case history. Consequently, the common misperception that an OT network monitoring solution from any OT network monitoring vendor can find subtle process sensor issues (e.g., sensor drift when the sensor is still in normal operating range, clogged sensing lines when the sensor is still in operating range, etc.) amply demonstrates the need for the engineering community to be involved.
I will be giving a presentation on process sensor issues (with SIGA) January 17, 2019 at the S4 Conference and preparing a paper on process sensor cyber security for the 2019 Texas A&M Instrumentation and Automation Symposium.
Joe Weiss
