There continues to be a lack of understanding about control system cyber security. What’s worse, there is a growing schism between network security/threat analysts and electrical, mechanical, control system, safety and other domain system engineers. This gap was laid bare in dozens of recent blogs discussing Presidential Executive Order 13920. It should be noted this is not just a US electric utility issue but is international in scope and affects industries beyond just electric.
Presidential Executive Order 13920 was not the result of a malware cyber event. Rather, it was the result of hardware backdoors that could not be detected by network security’s overly narrow focus on Ethernet and IP protocols. Network security’s narrow focus overlooks cyber attacks that take place beyond the focus on Ethernet and IP protocols, such as sensors, serial data streams, and protocols that are neither Ethernet or IP.
Apparently, the network cyber security community did not see the transformer issue or the Executive Order coming because neither are related to malware on the networks. Ironically it wasn’t a network threat analyst who saw this issue coming but Rebecca Smith from the Wall Street Journal. You can also find the April 28, 2019 blog on hacking transformers of interest - https://www.controlglobal.com/blogs/unfettered/large-electric-transformers-are-subject-to-cyber-attacks-which-can-cause-outages-of-months-to-years/ as it was evident since working on Aurora in 2011 that transformers could also be at risk to cyber attacks.
SANS prepared a 19 page report on https://www.controlglobal.com/blogs/unfettered/emergency-executive-order-13920-response-to-a-real-nation-state-cyberattack-against-the-us-grid/ in which SANS took strong exception to the blog post. In particular, they pointed to a lack of direct confirmation of concerns about a hardware vulnerability in the transformer supply chain. The SANS report was based entirely on a network assessment (see https://ics.sans.org/ics-library ICS Defense Use Case 7). Unfortunately, this wasn’t a network problem nor, as mentioned, was it detected by network security. The transformer issue was not the first time network security analysts overlooked non-Ethernet-based threats. The indirect evidence is disturbing and is well presented in Rebecca Smith’s Wall Street Journal article. The direct evidence comes from the site.
The Department of Energy (DOE) has yet to comment publicly on the inspection of either transformer. What has been released by DOE to date such as the DOE FAQs do not relate specifically to the transformer case. In the case of the transformers, the attackers were able to avoid the network security protections which is why it was not detected by network threat analysis or inspection. It is also why network security was outside the scope of the Executive Order. The lack of malware involved in this nation-state attack has the network threat analysis and security community in a tizzy.
Surprisingly (or maybe not), news reports in Chinese media. Global Times on June 3rd, claimed there is nothing to the Wall Street Journal’s story (https://www.globaltimes.cn/content/1190466.shtml). They claim it is just US disinformation in the trade war. Jiangsu Huapeng (the Chinese transformer manufacturer) categorically denies that its transformer was seized. The company says it was delivered to the end user in Colorado last summer and that Jiangsu Huapeng received payment in full. However, there are pictures of the first transformer installed at the substation in Colorado as well as the second transformer that was seized at the port of Houston and taken to the Sandia National Laboratory that discredits the Chinese claim. The transformer pictures and technical issues associated with the Executive Order will be discussed during the July 30th presentation to SURFA (see below).
The transformer issue was not the first time network security analysts overlooked non-Ethernet-based threats. Process sensors (e.g., pressure, level, flow, temperature, voltage, current, etc.) have no cyber security or authentication (see https://www.controlglobal.com/articles/2020/cybersecurity-for-field-devices/?utm_campaign=CGU_2020_Enews_Campaign&utm_medium=email&_hsmi=89865000&_hsenc=p2ANqtz-8Fba63D6MVTZGQAQt6NuXQf-WQpeCXU4tNSUD0FcPrpaZl1XFybaEowgU6zJInqfrfof0YpKllJIACzvh-WZFK2en0YA&utm_content=89865000&utm_source=hs_email) . However, the network security community generally refuses to acknowledge these gaps. The monitor and detect network technology generally overlooks breaches and other unintentional issues that occur outside of Ethernet/IP domain. Moreover, as mentioned in https://www.controlglobal.com/blogs/unfettered/the-connection-between-the-isa84-annex-h-on-process-sensor-cyber-security-and-presidential-executive-order-13920, process sensors can be used to compromise the transformers.
Everyone benefits when network security, physical security, and engineering experts work together. They did so with Stuxnet. However, the same can’t be said for defending control systems. Many attackers are aware of this gap and will develop their scenarios to attack where there is no monitoring such as the transformer case. The lack of understanding of the engineering issues was manifested in the 2017 Dragos whitepaper on CRASHOVERRIDE. The analysis addressed the remote opening of the breakers, not the reclosing of the breakers. Yet, the Aurora vulnerability and resultant long-term damage comes not from opening the breakers but from reclosing the breakers out-of-phase (https://www.controlglobal.com/blogs/unfettered/the-aurora-vulnerability-still-being-shunned-by-the-electric-industry-where-is-the-education/). This is an engineering issue, not common to network system analysts.
When incidents like the transformer issue arise, include the domain experts and you will get a better understanding of what can or cannot be expected to happen with the equipment. Relying on network security experts who focus on the Ethernet/IP band may be necessary, but it is certainly not sufficient. The SANS/Dragos transformer and CRASHOVERRIDE reports are clear examples. If the network security analysts work with domain engineers and technicians, the scope of such inquiries will be wider and more likely to identify and prevent unintentional incidents or malicious breaches.
There are T&D experts that could have been consulted but neither the Dragos report nor Honeywell’s Sinclair Koelemij’s Aurora blog appeared to do so. One T&D expert in particular was a substation manager for many years and involved in one of the only two Aurora hardware demonstration projects with DOD. He reviewed this blog, the Dragos report, and Sinclair’s blog. Suffice it to say, there are technical errors with the Dragos report and Sinclair’s blog.
Here are some other podcasts and presentations that may be of interest to those following the stories about grid security:
June 16, 2020, Civil Defense Radio posted an interview I did with Preston Schleinkofer May 13, 2020 on Presidential Executive Order 13920. As the interview predated the CSO and Wall Street Journal articles, there was no mention of either the name of the utility or there were two transformers involved. The interview can be found at Civil Defense radio - http://civildefenseradio.com/joe-weiss-on-electronic-control-systems-security/
July 15th 10amPacific, I will be giving a presentation on control system cyber security for the Purdue Cerias summer seminar series – “Cyber security of control systems – what needs to be done” (https://www.cerias.purdue.edu/news_and_events/events/security_seminar/summer)The focus will be on what makes control systems unique and will touch on the Executive Order.
July 30th 11amPacific, will be a panel session for the Society of Utility Regulatory Financial Analysts (SURFA) on the Presidential Executive Order 13920. Panelists will be Dave Batz from EEI, Phil Jones who was President of NARUC, and myself to address the technical issues and implications. This will be a very important session as these are the state regulators and utility financial analysts reporting to the Boards. The presentation will address real risk – system/equipment impact and physical consequences - not network vulnerabilities. Webinar details will be provided later.
Joe Weiss
