In the cyber security community, cyber vulnerability information sharing has been considered a critical need. Cyber information sharing has primarily been on network cyber vulnerabilities with both the suppliers and DHS providing cyber vulnerability disclosures. There is even a category to determine the severity of the cyber vulnerability. NIST’s Common Vulnerability Scoring Systems (CVSS) is an open framework for communicating the characteristics and severity of software vulnerabilities. However, the severity of the software vulnerabilities has not been translated into impacts on actual field equipment.
What is most important for control system cyber security is not the software vulnerability but the impact of the actual incident. This is because control system cyber security mitigation technologies and training need to be based on what is the impact that can be caused. However, cyber security technologies and training are based on network events not control system impacts. Control system cyber incidents can be unintentional or malicious. Either can cause significant damage and possible injuries though it may be difficult to determine whether the incident was malicious or unintentional. Control system cyber incidents can occur from older non-sophisticated vulnerabilities, equipment design features, modern cyber threats, etc. which may not even be addressed by the CVSS ratings. Moreover, CVSS scoring focuses on IT/OT-based environments while there are currently no known metrics that can be applied for control system device security vulnerabilities.
I have written about the lack of control system incident sharing though there has been little movement toward addressing it. The last time I did so, I got a response from the Ukraine on what had really occurred with the first grid cyber-attack which was different than what I had previously read. My non-public control system cyber incident database is now more than 1,250 and counting (there have been more than 1,500 deaths and more than $70Billion in direct damages). Many of these incidents would not have been identified or prevented by existing OT network security monitoring or cyber security policies. Some of the cases in my database came from giving presentations with example case histories where people realized they had experienced similar events. Identifying control system cyber incidents is difficult because in many cases there are no control system cyber forensics. Additionally, NERC continues to not identify control system cyber cases as being cyber-related. In other cases, the lack of control system cyber security training means that many “malfunctions” are not investigated as possibly being cyber-related as they are simply accepted as being “glitches”.
Several incidents have renewed the need to address actual control system cyber incidents. The examples include intrusions from both Russia and China.
The first incident occurred several years ago and was not intentional (though it could have been done maliciously). I found out about this incident from Linked-In where the Plant Technical Services Manager was asking for help as he was not getting answers from his control system vendor or system integrator. In this case with a multi-unit generating plant at power, someone from the IT network organization power cycled a network switch. The power-up of the switch initiated communication failures that propagated through the IT network into the plant Distributed Control System (DCS) and caused a loss of control system logic to more than 200 control system processors – a complete loss of view and control at the DCS with the units at power (In this case, the plant still had analog safety systems. If the plant would have had digital safety systems, there also would have been loss of safety). Plant operations was unaware of the power cycling of the switch or the impact it would have on the DCS. Conversely, IT was not aware of the impact they would have on the DCS and plant operations – and neither talked to each other. The network switch power cycling scenario occurred with several other power plants, including a nuclear plant, and a chemical plant with different DCS vendors and different network switch vendors. To date, there has been no disclosure or guidance offered to prevent this type of impact.
Recently, two very significant control system cyber incidents occurred that were not made public by DHS (it is not clear they were aware of either) and were not identified by the relevant industry ISAC (also not clear if they were aware).
The first case is the malicious hack of a drinking water facility. It bears some similarities to the 2000 Maroochyshire wastewater hack as a disgruntled ex-employee hacked into a control system. The Maroochyshire case was remotely opening sewage discharge valves to dump almost a million liters of sewage. Twenty years later, another disgruntled ex-employee hacked into a drinking water utility. I was made aware of this case as the ex-employee was caught and being sentenced. The defense attorney wanted to reduce the sentence by saying the defendant could not have caused any harm. The defendant accessed several pumping stations to change control system configurations and setpoints. I was called to concur that because the defendant was in the system for less than one hour, the defendant could not have caused harm. Suffice to say I could not say that (nor could the person called before me). If DHS or EPA were aware of the incident, one would have expected they would have provided this information to industry. However, neither the Water ISAC nor AWWA was aware of this incident. This was not the first time a water system’s control systems was hacked and the water industry not informed. In 2011, a water SCADA system had been accessed from Russia and a resultant Fusion Center report issued. Over a period of 2-3 months, minor glitches were observed in remote access to the water district’s SCADA system. The SCADA system powered on and off resulting in the burnout of a water pump, yet the case was made to “go away”.
The second case was a US pharmaceutical facility that had pharmaceutical equipment built by a Chinese company installed in a pharmaceutical manufacturing clean room. While doing maintenance on air handling equipment, a flow sensor was found that was not identified in the plant drawings. This led to a further detailed investigation where a complete “shadow” sensor network was discovered. The data was intended to be exfiltrated to China. It was not clear if the shadow system had control capabilities or was just to exfiltrate data.
Now consider Presidential Executive Order (EO) 13920 on the compromised large electric power transformer with hardware backdoors installed by the Chinese transformer manufacturer. Why is there so little public information on the incident and what has been found with the second transformer being examined at the Sandia National Laboratory? Why was there no information sharing about the “knock-off” Load Tap Changer from another US utility with a Chinese-made transformer - https://www.controlglobal.com/blogs/unfettered/ics-cyber-security-is-the-second-coming-of-the-maginot-line-and-the-chinese-have-breached-it? Instead of DOE telling industry what to do about the hardware backdoors in the more than 200 Chinese transformers already installed in the US electric grid, DOE has issued a request for information from industry. Because of the lack of industry guidance, many organizations view the EO as just another network cyber security problem – and it is not!
What are DHS, DOE, and others doing about disclosing information on control system cyber incidents? Why aren’t lessons learned from actual control system incidents being utilized in cyber security monitoring and control technologies as well as control system cyber security training programs? What does DOE expect the industry to do about the 200+ Chinese-made large electric transformers already installed in the US electric grid with unknown communication capabilities? Without control system cyber incident information, how can industry protect itself?
Joe Weiss
