Microsoft Windows has been widely adopted as a Human-Machine Interface (HMI) for Operational Technology (OT) networks which includes control systems, process sensors, and equipment monitoring. Why? Because it was there and available, not because it was optimized for the task. Windows has proven to be a great operating system for business systems and information exchange between Information Technology (IT) and OT organizations. But as an HMI to provide detailed engineering data, not so much.
Background
With the focus on the convergence of IT and OT, something more fundamental has been missed. Process sensors are the starting point for any physical process’s cyber security (not just industrial), product quality, reliability, and process safety. Process sensors provide operators with ground truth about their systems. However, process sensors have no cyber security or authentication.
Consequently, more accurate and authenticated process sensors can be expected to provide benefits across all aspects of the operational enterprise. To realize those benefits, and to take advantage of the promise of machine learning, a large industrial plant collected raw, unfiltered process sensor data on one of its manufacturing lines. This was a “typical” manufacturing plant with “typical” instrumentation, control systems, and plant equipment. The manufacturing line was selected because it had experienced ongoing issues with the reliability of the feed pumps, and the feed pumps unreliability directly affected the overall plant productivity. The manufacturing line included process sensors measuring pressure, level, flow, temperature, and valve position – typical of any industrial or manufacturing process. The HMI did not indicate any abnormal conditions. However, it was expected that plant productivity would improve as better understanding of the process sensors should contribute to improved pump performance and thus to improved plant productivity. Consequently, models were developed by Artificial Intelligence (AI) from machine learning inputs of the raw unfiltered sensor data. JDS Operational Technologies used the results of the machine learning to provide unexpected insights to the plant. Because the instrumentation, control systems, and equipment at that facility are common throughout industrial and manufacturing facilities, it is expected that these insights can be extended to all operational sectors.
Windows
Windows was developed as a general-purpose operating system for consumer and “front-office” applications, not as a data acquisition tool for “back-office” engineering applications. That is, Windows was not designed for monitoring physical processes or control system devices in “near” real-time (milliseconds to seconds).
Data acquisition
Data acquisition is accomplished by scanning the field inputs. Once the data had been acquired by the field equipment, it is sent to the HMI where the HMI software scans the acquired data (the scan rate). The scan rate is a measure of the response of the HMI to changes in field conditions. A slower response reduces the ability of the control system to automatically respond and provide operator information to system changes. The Windows scan rate issues have affected the ability to identify abnormal situations in numerous facilities such as the inability for early identification of plugged sensing lines, flow-induced vibration, pump cavitation, etc.
Plant data analysis
The plant HMI was Windows-based, though the slow scan rate issue is not unique to Windows. The assumption was the HMI was a faithful representation of process sensor/system operation. That is, the operator should trust the data displayed by the HMI. Machine learning was used on the raw unfiltered process sensor data (no scan rate limitation) to evaluate whether there was additional data beyond that in the HMI that could be of value in improving the pump performance.
The plant test results demonstrated that the HMI did not detect “fast” system or process sensor behavior (seconds or less), events like rapid sensor oscillations—exactly the type of high frequency noise that engineers are interested in detecting and resolving. The operator needed to know which trends in the data to monitor. Even with the HMI tied to modern data analytics tools, the HMI did not provide any indication of off-normal trends in the plant data. However, this was inconsistent with the data from the machine learning from the raw unfiltered process sensor signals which indicated that more than half of the sensors were either inoperable or out-of-calibration. Additionally, the feed pump experienced upsets that were less than one minute in duration that the HMI did not detect.
Impacts
Generally, plant productivity improvements are made by upgrading or adding larger plant equipment. However, those types of upgrades are very costly and are also dependent on process sensor accuracy. In this case, process sensor issues significantly affected plant productivity. The sensor/equipment impacts identified from this test were on the order of a 2-3% reduction of overall plant productivity resulting in significant financial losses. Consequently, improved productivity can be made by without installing new plant equipment by using machine learning of the process sensors and the process.
Process sensor issues also affected cyber security. As mentioned, process sensors have no inherent cyber security or authentication. This means that the OT monitoring system is protecting untrusted data (no authorization or authentication) of unknown quality. It also means operators are taking actions based on data of questionable validity. In this case, using the raw data directly from the process sensors inherently provided authentication – a cyber security improvement. The raw sensor data being collected and analyzed off-line from the OT network also provided additional layers of cyber security including from IT malware such as ransomware. Machine learning also provided more meaningful and correct data to the operators.
Summary
Process sensors are the input for predictive maintenance, digital transformation, Industry 4.0, smart manufacturing, smart grid, etc. The majority of OT networks use Windows-based HMIs even though Windows was not designed to be an engineering data acquisition tool. As such the HMI was not effective and, in fact, provided misleading information on the state of the process sensors and plant equipment. Using raw unfiltered process sensor monitoring and machine learning, the plant now has the potential to demonstrate a significant return on investment (ROI) from improved plant operation as well as improving cyber security protection. Machine learning also provided more meaningful and correct data to the operators. Monitoring tools for process sensors and plant equipment need to be purpose-built, not general-purpose systems such as Windows. As the test data is still being analyzed, more details will be included in future articles including in the November issue of IEEE Computer magazine.
Joe Weiss
