Abstract
This blog could have been written by the famous threat analyst Pogo Possum who said: “we have met the enemy and they is us”. Presidential Executive Order (EO) 13920 was meant to prevent the use of Chinese-made products in critical bulk electric grid applications. The EO was also meant to address hardware supply chain issues as the Chinese had installed hardware backdoors that bypassed all cyber security protections. However, neither is happening. As a result, China is in a position to “own” our electric grids and other critical infrastructures at times of their choosing.
Background
Cyber security of the control systems in critical infrastructure was started by Presidential Decision Directive (PDD63) in 1998. However, the national discussion on cyber security has focused on data breaches especially ransomware which precipitated the August 25, 2021, White House Cyber security meeting. For IT networks, the focus on data breaches may be sufficient. However, the concern for critical infrastructures is not data breaches but equipment damage that can cause extended outages and kill people. Until the 2019 discovery of the hardware backdoors in the Chinese-made transformer installed at the Western Area Power Administration (WAPA) Ault substation outside Denver, Chinese cyber exploits were about stealing data. However, the installation of backdoors in the WAPA transformer was about taking control of the transformer as there was no data to steal.
There are cyber forensics for Internet Protocol networks. However, there are no cyber forensics and often limited cyber security training for the operational engineers that design and operate the control system field devices (e.g., process instrumentation, actuators, drives, analyzers, etc.) and control system serial networks that cause equipment damage not data breaches. It is these control system field devices that can go “boom in the night”.
Chinese-made transformers
Recently, Michael Mabee issued the following FERC Complaint including the exhibit A – bills of lading for Chinese-made grid equipment https://michaelmabee.info/chinese-transformer-complaint-filed-with-u-s-government/. Prior to 2020, there were approximately 200 large Chinese transformers in the US electric grid. According to the Chinese transformer manufacturer JSHP, their Chinese transformers supplied 10% of the power to New York City and almost 20% to Las Vegas. Chinese transformers also supply power to some very sensitive locations.
EO 13920 was issued because of the hardware backdoors found in the Chinese-made transformer installed at the WAPA Ault substation outside Denver in August 2019. EO 13920 was meant to reduce or eliminate the use of Chinese-made equipment in the US electric grid. However, according to the US International Trade Commission, there have been 54 large Chinese-made electric transformers delivered to the US electric utilities in 2020 with more on order in 2021. Additionally, EO 13920 has been transformed by DOE and the electric industry into a software supply chain problem. Software bills of materials (SBOMs) are of little value when the equipment is made in China.
Another large Chinese-made transformer was sent to the Sandia National Laboratory for detailed evaluation. The US Department of Energy (DOE) has not shared information on what has been found with either the WAPA transformer or the transformer at Sandia. However, recently, the presence of hardware backdoors in the Chinese-made transformer at Sandia was confirmed by Latham Saddler (https://www.youtube.com/watch?v=x0EawFC18MI) who was the Director of Intelligence Programs at the National Security Council during the Trump administration. Latham stated that the Chinese transformer was taken to “one of our national labs,” and something was found in the transformer: “They found hardware that was put into the transformer that had the ability for somebody in China to switch it off.” Not only has this information not been shared with US utilities, this information has not been shared with our closest allies who also have these Chinese-made transformers.
Given that hardware backdoors were found in these two Chinese-made transformers, what does that mean to the other almost 300 Chinese-made transformers already installed in the US grid?
There are a variety of concerns about the Chinese transformers. They are, for example, approximately 20-30% less expensive than corresponding US-made transformers. Consequently, US utilities keep buying them. Additionally, determining the presence of hardware back-doors and potential manufacturing “defects” are very difficult to detect.
Use of Chinese-made grid equipment
Michael Mabee has amassed more than 150 bills of lading for Chinese-made electric equipment exported to the US electric grid from 2018-2020 (see Appendix A of Michael Mabee’s submittal). Utilities using this Chinese-made electric equipment range from the relatively small (e.g., the City of Anaheim, the Grand River Dam Authority) to the large (e.g., PG&E, Public Service New Mexico, WAPA, BPA, Iberdrola, MidAmerica, Pacificorp, Talen Montana, Florida Power & Light, NV Energy, Nebraska Public Power District, LADWP, SMUD, EDF Renewables, and New York Power Authority-NYPA) and grid equipment suppliers (e.g., Alstom Grid). Some of this Chinese-made equipment has been installed in very sensitive locations.
DoubleTree Systems, Inc. (http://www.dsius.com/) is associated with JSHP transformers and other Chinese equipment manufacturers connected to the Chinese government. Double Tree Systems continues to provide critical grid equipment and engineering services, including equipment explicitly addressed in EO 13920, to US utilities. Double Tree Systems not only imports and markets Chinese JSHP transformers in the U.S., but sells a variety of critical grid monitoring products and services:
- POLARIS (substation monitoring system)
- SA200 (substation automation)
- Wide Area Measurement System (WAMS). According to Double Tree System’s website: “WAMS solution provided is field-proven in Bonneville of Power Administration of WSCC (sic)”.
- Generator Testing & Model Validation. According to Double Tree System’s website: “The standard generator testing and model validation provided by Double Tree Systems, Inc. has extensive experience and has been certified by Western Systems Coordinating Council (WSCC)”.
- Special Protection System (SPS).
- Transfer Limits Monitoring.
- SCADA/EMS/DMS/DA consulting.
The equipment and services provided by Double Tree Systems through POLARIS, WAMS and SA200 and the companies associated with the Xu Ji Group Co. and JSHP should raise major concerns about the integrity of the US electric grid. These grid monitoring and control systems can help China set up covert cyber infrastructure that can compromise the nation's electric grid. For example, these systems can impact the accuracy of the phasor measurements being counted on to provide grid monitoring and system restoration forensics.
The Electric Power Research Institute (EPRI) did a demonstration program of the Chinese-made grid equipment. EPRI’s write up on the 2014 DISTRIBUTCH Conference and Exhibition noted: “Network Security: A kickoff meeting was held on the floor of DistribuTECH for a group of vendors currently interested in participating in the ‘Protective Measures for Securing T&D Systems’ project, which involves validating the mapping of IEC 62351‐7 network security events. Vendors currently engaged in the project include SISCO, Ruggedcom, OSIsoft, Double Tree Systems, and Radiflow. Other vendors expressing interest include Cisco, Schneider Electric, and SEL. The project will be driven by use cases developed in early 2014 with proof‐of concept implementations to be developed and demonstrated in the EPRI Cyber Security Research Lab throughout 2014.” Not only does this project provide additional justification for US utilities to use the Chinese-made equipment, it also enables US electric grid suppliers to use this Chinese equipment.
Possible mitigation
DOE is developing technologies to detect electric grid cyberattacks such as C3D (https://www.controlglobal.com/blogs/unfettered/results-from-threatconnect-webinar-on-mitigating-risks-in-critical-infrastructures-and-on-going-actual-risks). The C3D technology does not address hardware issues, nor do they address known issues such as Aurora. Additionally, C3D can utilize data from Double Tree Systems products as input. With the Chinese-made equipment and services, the lack of implementing Aurora hardware protection, and the gaps in protection inherent in the NERC CIPs, one has to ask how is it possible to claim the grid is being protected against cyber threats?
The sensor monitoring technology identified in https://www.controlglobal.com/blogs/unfettered/us-critical-infrastructure-cyber-security-is-backwards-its-the-process-that-counts-not-the-data/ can help by providing authentication of the process sensors to help minimize the impact of the hardware backdoors and man-in-the-middle cyberattacks. However, the NERC Critical Infrastructure Protection (CIP) standards and the NERC Supply Chain requirements exclude monitoring of the process sensors. This irrational exclusion has to change. The process of adding the requirement for process measurement integrity to the appropriate International Society of Automation (ISA) 62443 series of control system cyber security standards was raised as data integrity was already included, but process measurement integrity was not. In fact, there is even a need to define the term “process measurement integrity”. If you can’t trust what you measure, there is no cyber security, safety, or resilience.
Summary
Chinese-made grid equipment is widely used in the US electric grid. Backdoors have been found bypassing all cyber security protections that can cause grid equipment damage and wide spread grid disturbances. Securing the electric grid will require a combination of many different approaches including making the economics of buying American-made products more affordable (or making Chinese-made products less affordable), eliminating use of known Chinese-front companies providing grid equipment and services, addressing networking and engineering issues, changing the scope of the NERC CIPS to focus on the reliability and cyber security of the grid not just routable networks, monitoring the process sensors off-line and in real time, and having engineering participation.
Joe Weiss
