Process sensors are ubiquitous and are under the auspices of the engineering organizations. Process sensors are the input to control and safety systems and provide input for operator decisions. Like our fingers, eyes, and ears that provide input to the brain to make the right decisions, if the process sensor input is not secure and accurate, catastrophic failures can occur. This has often meant that attention to safety, which the engineering organizations understand, has outrun attention to security, which is something that the engineering organizations have tended to view as the responsibility of the IT organization. And in turn, the IT organizations have tended to overlook process sensor security, seeing that as an engineering responsibility that’s outside their own scope. At the process sensor level, however, safety and security are really the same issue.
Interested industry representatives meet to discuss process sensor security and safety
On December 14, 2021, I gave a Tech Talk to the IEEE Consultant Network Seattle Affinity Group of Seattle Section. As a result of that presentation, IEEE’s Sheree Wen expressed an interest in identifying and addressing standards organizations’ gaps and vulnerabilities associated with the cyber security and resiliency of control system field devices, including process sensors. Consequently, a virtual meeting was held January 5th, 2022 under the purview of IEEE, with universities and standards development and industry organizations representing a cross section of critical infrastructures. Essentially, “a coalition of the willing”. As the process sensors are used in all sectors, the intent of the meeting was to create outcomes and a way forward for advancing cyber security and reducing risk associated with insecure process sensors in a common manner.
The Challenge of securing process sensors
Some example process sensor cyber-related incidents include:
- Dam collapse from erroneous low-level readings
- Sensor malfunctioned resulting in the release of 10 million gallons of untreated wastewater
- Safety relief valve in a nuclear plant did not lift because the pressure sensor never reached its setpoint
- One voltage sensor failure in combined cycle plant in Florida caused a 200MW load swing at the plant that resulted in a 50MW load swing in New England
- Tank farm explosions from erroneous level sensor readings
- Airplane crashes from erroneous sensor readings
- Refinery explosion from erroneous sensor readings
There was a linked-in note on January 5, 2021, that the first ever cybersecurity standards specifically for building control systems was issued built on the ISA/IEC62443 series of standards. September 2021, the Oak Ridge National Laboratory (ORNL), Pacific Northwest National Laboratory (PNNL), and National Renewable Energy Laboratory (NREL) issued the report Sensor impacts on building and HVAC controls: A critical review for building energy performance. According to the report, “Cybersecurity threats are increasing, and sensor data delivery could be hacked as a result. How hacked sensor data affects building control performance must be understood. A typical situation could include sensor data being modified by hackers and sent to the control loops, resulting in extreme control actions. To the best of the authors’ knowledge, no such study has examined this challenge.”
Process sensors have no cyber security, authentication, or cyber logging. Consequently, it is not possible to know whether these incidents were intentional or malicious but made to look like they were unintentional.
There are three questions that are often asked about cyber security of process sensors:
- Do you need a physical presence to compromise the sensor? No, it can be done remotely.
- How much harm can cyber-related sensor impacts cause? The field calibrator calibrates one sensor at a time but connects insecurely to the Internet. The Asset Management Systems (AMS) has access to thousands of sensors. Meanwhile, the AMS has insecure connections to the Internet and often is connected to the Corporate Enterprise Resource Planning (ERP) systems. Some real examples of catastrophic failures from sensor issues were provided.
- What happens when the compromised sensor data is sent to the cloud to be used in big data analytics for IOT or Industry4.0 applications? The sensor data is assumed to be uncompromised.
These deficiencies lead to a need for a training environment to:
- Better understanding of how an adversary may interrupt, degrade, or possibly damage and destroy infrastructure.
- Develop forensic capability to detect process sensor cyber-related issues.
It should be noted that an appropriate training facility would accomplish the above tasks whether the sensor issues are malicious or unintentional.
Process sensor security may amount to a gap in standards and regulation
Three discoveries and events the week prior to the meeting elevated the groups concerns.
- Based on discussions with the Transportation Security Administration (TSA), the recognition that the TSA pipeline cyber security guidelines did not address control systems including the sensors.
- The “discovery” in Abu Dhabi that more than 3000 sensors had no ability to have passwords https://www.controlglobal.com/blogs/unfettered/a-vulnerability-worse-than-log4j-and-it-can-blow-up-facilities-and-shut-down-the-grid/.
- The recent API 1164 pipeline cyber security guidelines (August 2021) effectively excluded the process sensors (Clause 6.6.5.4 (b) "Inventory should not include individual instruments that are not network connected".
As the cyber insecure sensors in the Abu Dhabi petrochemical plant show, digital sensors have built-in backdoors for performing remote calibration and other maintenance activities. That makes sense as a convenient, labor-saving design feature. It makes the sensors easier to upgrade and maintain. These same backdoors, however, can be exploited as vulnerabilities, even when the sensors do not appear to be connected to networks (see the problem with the API standards). In essence, the backdoor in the process sensors allows for two-way communication to/from the Internet with no cyber security protection. An indication of the disconnect between engineering and cyber security is that many engineers would be willing to pay extra to have the backdoors because it makes their jobs easier despite the cyber risk. The same cyber vulnerabilities in the process sensors also exist for the field calibrators and the AMS. At the 2016 ICS Cyber Security Conference, both the U.S. Air Force Institute of Technology (AFIT) and the Russians demonstrated how process sensor cyber vulnerabilities could be exploited. The Russian demonstration exploited the cyber vulnerability in the AMS while the AFIT presentation addressed the cyber vulnerability of multiple process sensors.
Organizations attending
The January 5th discussions were held under Chatham House rules so there were no names or attribution. The meeting included representatives from numerous industry, non-profit, and academic institutions:
American Gas Association (AGA)
American Society of Mechanical Engineers (ASME)
American Petroleum Institute (API)
American Water Works Association (AWWA)
FieldComm Group
International ElectroTechnical Commission (IEC)
Institute Electrical and Electronic Engineers (IEEE) changed name to ‘Advancing Technology for Humanity’
International Congress on Systems Engineering (INCOSE)
International Society of Automation (ISA)
Food and Agriculture (Infragard)
American Bureau of Shipping (Maritime)
Mining and Metals
MITRE
Pharmaceuticals
National Fire Protection Association (NFPA)
National Commission on Grid Resilience
Society of Automotive Engineers (SAE)
Utilities (electric, water, energy)
University of Indiana
University of Texas-San Antonio
Washington University
Progress, but much work remains
The participants recognized that considerable progress has been made for control system (Operational Technology-OT) network security. For example, guidance now exists with the ISA/IEC-62433 standards, NIST SP800 series standards, and various other guidance documents. However, these standards do not yet address cybersecurity considerations at the lower levels of the Purdue Reference Model, namely Levels 0,1 process sensors/devices and field sensor networks. The group concluded that additional research, training, and testing to improve process sensor cyber security is lacking and requires new and innovative efforts from both industry and government leaders.
The discussions must be trans-disciplinary and must include engineers and facility operators as well as IT and OT networking personnel. The numerous actual control system cyber incidents clearly demonstrate that current approaches fail to sufficiently consider the engineer/operator roles and responsibilities in identifying and mitigating threats. There is consensus that engineers’ contributions to security and resilience will be stifled if the perception continues to be viewed that control system cyber security is just network and IT/OT problems. One of the attendees noted “this is less about cross-industry than it is about cross-discipline. Limitations in one discipline (e.g., instrumentation-sensors) can lead to vulnerabilities in another like security.” Consequently, this is not industry or sector specific. It requires the cooperation between physical security, network security, and engineering/operational security disciplines which can be fostered and enhanced via collaboration between professional associations and societies like IEEE, ISA, ASME, etc. To summarize, the networking community currently dominates cyber security and views all sectors as effectively being an extension of IT (https://www.controlglobal.com/blogs/unfettered/ot-network-security-often-does-not-view-control-system-devices-and-the-process-as-their-problem). Meanwhile, the engineering community has limited participation in cyber security decision making process as the engineering equipment that is often vulnerable is ignored by networking cyber security (https://www.controlglobal.com/blogs/unfettered/engineering-operations-and-maintenance-often-do-not-view-cyber-security-as-their-problem). A cross-disciplinary approach represents an important first step in bringing the engineering discipline to help address the cyber security of control systems which is generally not done when the focus is just securing the networks. To demonstrate the feasibility of a cross-discipline approach, a mining project in Canada was discussed that was using raw process sensor monitoring for productivity and maintenance improvements. As cyber security was also involved, the project brought multiple organizations together - corporate, plant engineering, operations, maintenance, safety, and cyber security. This project demonstrates that a cross-discipline approach is possible (in fact, necessary).
The group identified there are two distinct categories of process sensors to be addressed:
- Legacy devices - These are the devices currently in use and those still being built. There is no cyber security in these devices or cyber security standards to address these device limitations.
- Nextgen devices - Nextgen is still “on the drawing board”. ISA/IEC62443-4-2 can addresses these devices. However, at today’s funding level, Nextgen is arguably years from a prototype.
Historically, the network community has questioned whether process sensors should be within the scope of cyber security efforts. They question if process sensors are computers and if they are on networks. Process sensors may not look like computers, but they have similar components such as microprocessors which perform familiar computing functions. Sensors are also on networks, often serial as opposed to routable networks. The confusion may arise because many in the networking community view networks as being routable networks and therefore don’t recognize serial networks as being networks. This can be seen in the NERC CIP standards which only recognize routable networks.
A way forward for process sensor security
The group concluded that establishing standards and guidelines to address the unique gaps and vulnerabilities with legacy field devices remains a priority. Despite the value of tools such as the MITRE ATT&CK tool and the CVE methodology, more work is needed. The MITRE ATT&CK tool doesn’t address process sensors and other control system field devices. This needs to be added based on actual cases. The CVE methodology for software vulnerabilities has no counterpart for control system hardware. This also needs to be added.
Participants felt that consideration should be given to establish a Sensor STIG (Security Technical Implementation Group). There should be some sort of threaded discussion board where discussions could continue after this meeting. There was also the question as to what organizations are best suited to sponsor and oversee this new and expanding area of training and research for control system field device cyber security.
The team will develop a White Paper to be shared with government policy makers and R&D organizations who are able to resource and facilitate these efforts The white paper will clearly define what is unique about legacy control system field devices and what needs to be done to provide improved cyber security as funding is necessary to expedite developing cyber security standards, frameworks, recommended practices, and information sharing for this inter-sector community.
Historically, standards have been driven by industry. It’s time for the industry that relies on process sensors to take the lead in closing the gap between cyber security and safety engineering.
Joe Weiss
