With the annual RSA Cyber Security Conference next week in San Francisco, I thought this would be an opportune time to discuss the validity of cyber security policies for control system environments.
Cyber security policy has been based on preventing malicious attacks against IT data networks. IT cyber security has been a problem for more than 20 years. With all of the money and attention being paid to IT cyber security, it is still far from being a solved problem. Meanwhile control system cyber security is arguably more than 5-10 years behind IT security with much less management attention and associated funding (see https://www.controlglobal.com/blogs/unfettered/a-retrospective-on-the-first-two-decades-of-control-system-cyber-security-culture-issues-still-prevent-successfully-securing-control-systems/).
Control system cyber security is meant to keep “lights on,” “water flowing,” pipes from breaking,” “trains from crashing,” etc. Some of the more important definitions used for control system cyber security are very different than the same terms used for IT. These include the definitions of “endpoints”, “cyber incident”, and “Operational Technology-OT”.
- Endpoints - For IT, endpoints are firewalls, routers, switches, cell phones, etc. For control systems, endpoints are Level 0,1 devices including process sensors, actuators, and drives.
- Cyber incident - For IT, a cyber incident is a malicious attack to steal or change data. For control systems, it is the NIST definition of electronic communications between systems that affects Confidentiality, Integrity, or Availability (note that the term Safety is missing). What is important is the term “malicious” was intentionally not included because a control system cyber incident can be malicious or unintentional as the impact can be the same.
- OT- For IT, the term OT is everything that isn’t IT. For control systems, OT are the people responsible for the OT networks. Generally, OT is not the engineers and technicians responsible for turbines, pumps, motors, relays, instrumentation, or safety systems – that is engineering.
Additionally, there are three key terms that often are used synonymously but are not the same: “reliable”, “safe”, and “secure”. You can be secure but not safe; you can be reliable but not secure, etc. Until, the IT/OT and engineering organizations can actually talk to each other, the vast culture gap will remain (see governance issues below).
Control system cyber impacts are real. There have been more than 1,200 actual control system cyber incidents to date, with more than 1,500 deaths and more than $70 billion in direct damages. The impacts include pipe ruptures, refinery explosions, train crashes, plane crashes, major electric outages etc. all of which were caused by electronic communications between control systems. As there are minimal control system cyber forensics and sophisticated attackers can make cyber attacks appear to be equipment malfunctions, it is difficult to identify which incidents are unintentional and which are malicious.
Because many control systems use IP-based OT networks and Windows-based Human-Machine Interfaces (HMIs), cyber security policy makers have assumed that control systems are simply another type of IT infrastructure, and therefore IT policies, technologies, training, and testing methodologies apply. This is not true. There have been numerous cases where IT security policies, procedures, technology, and/or testing have impacted control system operation and in some cases have actually damaged control systems.
Cyber security is generally under the purview of the CIO and/or CISO. Yet, in most cases, these positions have no responsibility for the design, procurement, operation, or maintenance of control systems. However, the VPs who manage power production, power delivery, manufacturing, refining, buildings/data centers, etc. generally have no input or participation in the cyber security of these systems that affect their responsibility. Simply trying to get people from engineering and IT/OT to work together on their own, “donut diplomacy”, has not worked. Examples of this broken governance model include:
- Multiple cases of control systems being shut down or processes impacted because the IT/OT organization didn’t understand the impacts of their actions on the control systems but wouldn’t talk to the engineers to resolve the potential problems.
- From an e-mail to me from an automation specialist working on OT: “Lots of companies don't have security specialists with expertise in industrial automation. Moreover, most of the industrial automation specialists don't tolerate IT security specialists and try to keep a distance and what is worse even impede the implementation of security controls.”
It should be evident that a governance model for control system cyber security that does not explicitly call for engineering and IT/OT to work together is doomed.
As mentioned, control systems generally consist of OT networks, Windows-based HMIs, as well as control system devices such as process sensors, actuators, drives, power supplies, etc. OT network cyber security share aspects of traditional IT cyber security. However, control systems are different than IT/OT networks in many crucial ways policy makers need to understand. Clearly, they need, appropriate policies, procedures, training, and monitoring:
- Control systems are designed for reliability, safety, efficiency, and are “open” systems that rely on trust. The IT “zero trust” model does not work for control system devices.
- Level 0,1 devices are resource-constrained and have no ability to implement cyber security, cyber logging, or device authentication. There are tens of millions of unsecurable process sensors and other control system devices in all industries, transportation systems, buildings, etc. As “Mother Nature” operates in the analog world, control system cyber risks begin with the unsecurable analog measurement before OT network monitoring even comes into the picture. Consequently, the analog measurement is part of IIOT, Industry 4.0, Smart Grid, and all other forms of digital transformations, yet is generally ignored.
- Control systems are deterministic, meaning that delays, like implementing encryption, cannot be tolerated without causing a denial-of-service.
- Control systems operate automatically, often in milliseconds, where an operator cannot intervene. Moreover, there are control systems that don’t even have an HMI in the familiar sense of the term.
- Control systems are designed for many years to decades of service, and realistically will not be replaced due to cyber threats.
- Control systems have design features that can be cyber vulnerable yet cannot be changed nor can IT policies affect them as they are part of the equipment design. Examples include hardcoded default passwords and built-in back doors. It will require appropriate policies, procedures, and training to work around these types of built-in vulnerabilities as these issues will be present for the next 10-15 years.
- Control systems often cannot be patched or modified while in service, which can lead to months to years of operation without patching. Additionally, inappropriate patching, can and has, shutdown or damaged control systems.
- Cloud computing erroneously assume all process sensor input are uncompromised, authenticated, and correct. Cloud computing providers have no ability to add security to insecure sensor input.
Cyber attacks can be difficult to recognize even in familiar IT networks. The problem is even tougher with control systems. Sophisticated cyber attackers will make cyber attacks look like equipment malfunctions. As an example, Stuxnet was destroying centrifuges for a year before it was found to be a cyber attack. Until that time, the damage to the centrifuges were thought to be from a systemic design flaw. The designers, operators, and maintainers of control system equipment are engineers and technicians, not computer experts. There is minimal, if any, control system device cyber forensics. Because cyber security training has been focused on OT network not control system personnel, many of these types of attacks cannot be detected by network monitoring and the engineers are untrained to recognize if equipment malfunctions are actually cyber-related. This culture and training gap between networking and engineering starts in the university setting where engineering and computer science/computer security are often not connected. This culture gap between the networking and engineering domains need to change along with the governance model.
Most importantly, control systems “manipulate physics” not data meaning compromise of control systems can, and has, led to catastrophic physical failures. This can be existential for industrial or manufacturing facilities as well as for our society. Consequently, the unique aspects of control systems need to be adequately addressed by policy makers but have not. Developing inappropriate cyber security policies can be counterproductive at best. Consequently, it is imperative to have the right expertise involved when developing cyber security policies that can affect control systems and societal good. Yet, most control system cyber policy conferences have minimal to no participation from control system cyber experts.
All of this means that cyber security policies and procedures need to be developed specific to the unique aspects of control systems which is why the International Society of Automation (ISA) is developing the ISA-62443 series of standards. Other technical organizations such as the International Council on Systems Engineering (INCOSE) and the Society of Automotive Engineers (SAE) with control systems expertise are also involved. However, there are a multitude of other organizations developing standards and guidelines that can affect control systems that may be inconsistent and not address the control system-unique issues. These efforts must be better coordinated.
Recommendations:
- Governance changes need to mandate that engineering management/technical staff participate in control system cyber security along with CIO/CISO and IT/OT networking staff. This includes having cyber security policy meetings include control system cyber security experts. Provide cyber security training to the control system engineers to help identify if equipment malfunctions could be cyber-related. Treat control system cyber incidents as business continuity issues and have appropriate trained personnel available for manual operations.
- Recognize that the most dangerous control system cyber attacks are those that manipulate physics such as the Aurora vulnerability. This requires engineering domain expertise to identify and respond to these types of issues.
- Provide R&D funding for securing Level 0,1 devices and networks.
- Universities need to require introduction to computer science/cyber security for engineering students and computer science/computer security students to take an introductory course on engineering.
- Have control system cyber security and process safety standards adequately address Level 0,1 devices.
- Coordinate control system cyber security standards and regulatory activities.
It is imperative that cyber security policy makers take into account the unique technical and policy/governance issues associated with control systems. Current cyber security policies often have made control systems more susceptible to unintentional or malicious attacks because of the lack of input from the engineering organizations. This culture/governance gap needs to change.
Joe Weiss
Leaders relevant to this article: