Why would attackers hit defenses head-on when they could simply bypass them? The two transformer hardware cases along with the backdoors in smart transmitters led me to the epiphany that OT cyber security is like the Maginot Line from World War II. The focus for ICS cyber security has been on the OT networks, assuming all OT cyber threats have to go through the OT Ethernet networks where they could be detected and hopefully blocked. In the WAPA case (https://www.controlglobal.com/blogs/unfettered/presidential-executive-order-13920-was-not-due-to-a-malware-event-recent-and-upcoming-events-will-discuss-the-event) and the load tap changer (LTC) case (this blog), the Chinese went around the network cyber security and bypassed the OT cyber security Maginot Line. This occurred to me while I was going over slides I was preparing for the July 30th SURFA meeting. An individual familiar with the JSHP transformer case at WAPA reviewed the slides with me. He had mentioned that there was a knock-off LTC Load Tap Changer (LTC) in a Chinese transformer. I assumed he meant the knock-off LTC was at WAPA and included that in my draft slides for his review. However, my contact told me it was NOT at WAPA but at another utility, this one investor-owned.
The Chinese hardware backdoors reminded me of two instances of particular significance. In the mid-1990s when I was managing the EPRI Fossil Plant Instrumentation & Controls (I&C) Program, we had a meeting of the I&C Advisory Committee at the Westinghouse Process Controls Division where they made the Westinghouse plant Distributed Control Systems – DCSs (the Westinghouse Process Control Division was subsequently bought by Emerson). A large number of the systems on the factory floor were on their way to China for steel, power, water, etc.
The second instance was in 1999 when I attended the joint US/China Y2K Conference in Shanghai. Following the Conference, I was given a visit to a new multi-unit coal-fired power plant. I thought I recognized the equipment in the plant as it looked just like an ABB-CE coal-fired plant with Westinghouse turbines I had seen at many US power plants. However, I was in for a surprise. Except for the first-of-kind Max DCSs, everything in that plant had Chinese nameplates, including what I thought were the Westinghouse turbines and turbine controls, as they were obviously reengineered. The Chinese certainly know our systems. As Sinclair Koelemij in his blog 7/4/20 blog – “Dare for More, featuring the ICS kill-chain and a steel mill” discusses the 2 steps of the cyber kill chain. Step 1 is reconnaissance and information gathering while step 2 is the attack. When you supply the equipment, there is no need for Step 1.
The original intent of ICS cyber security was to keep lights on, water flowing, and maintain reliability and safety. Unfortunately, the intent of ICS cyber security has been changed by the OT network security community and it is no longer about preventing system impacts but about maintaining OT network integrity. As Dragos’ Rob Lee told me about process sensors, why care about the sensors when everything has to go through the OT network and the OT network threat hunters and network security will detect any anomaly (this is a common mantra from the OT network monitoring vendors). In the Second World War the Maginot Line was an effective, advanced fortification, and the invading German forces simply bypassed it rather than try to breach it. Similarly, OT network defenses are also effective and advanced, which is why attackers have an incentive to bypass them. This is particularly concerning, considering that compromised sensors could cause equipment issues at the power system level since control systems rely on accurate input that agrees with potential transformers (PT) and current transformers (CT) inputs. Unfortunately, this is the reason I feel OT network threat hunting and cyber security is the second coming of the Maginot Line. The Chinese have already demonstrated they know our equipment, have similar knock-off equipment, and can get around the “Maginot Line” by installing hardware backdoors. The Chinese have done this with at least one transformer in the US. It is unclear whether the other 200+ large Chinese transformers installed in the US grid have similar backdoors which is the reason for Presidential Executive Order 13920. As a result, the Executive Order excludes the Maginot Line (network monitoring) as it has been bypassed by the hardware backdoors. Now we have another case that may be even worse. These are not the only cases where the OT Maginot Line can, or has been, breached. As mentioned in previous blogs, smart process sensors have built-in back doors that cannot be closed. There are also other examples.
First some background about LTCs. Transformers are the devices that transform the voltages from higher voltages to lower voltages or lower voltages to higher voltages. This is accomplished through the design of the coils (windings) in the transformer, and is a function of the turns ratio – the number of turns in one winding relative to the number of turns in another. When supplying power to a customer load, the transformer output must be kept at a relatively constant voltage level regardless of the changing current required by the load. This is accomplished by installing an LTC onto the transformer. The LTC changes the transformer turns ratio by moving among a number of access points or “taps” on one of the two windings, thus either raising or lowering the voltage supplied to the customer. This can be done manually or automatically, although both rely on the measured voltage. If the voltage measurement is compromised, the LTC can be at risk in either manual or automatic operation. If an LTC fails, the entire transformer will be out of service. In extreme cases where the transformer cannot be disabled, the LTC will be manually locked to a particular position as determined by the engineers. If an outage is necessary, this will have an adverse effect on numerous transmission or distribution circuits and an adverse effect on the remaining power grid due to the need to reroute the load to supply the affected circuits. Should the transformer be one of two transformers designed for a parallel transformer design, which is common in transmission stations, then the implications become far more dangerous because of the introduction of circulating current between the two electrically connected transformers. This could lead to severe physical damage to the transmission station and the transformers themselves as well as the potential danger posed to any utility personnel within the station itself.
Buying a large transformer is like buying a commercial airplane. You are essentially buying the frame from Boeing or Airbus and then specifying whose engines, flight electronics, etc. should be included to make the final airplane. Buying a transformer is similar. You procure the “transformer box” and then separately specify the LTC, protective relays, transformer diagnostics, etc. to provide a complete transformer system. The specification of a transformer for a generation or transmission station is extremely detailed and the inspection and acceptance testing is thorough in its requirements. The transformer is a major component of those stations and accounts for up to half the cost of the stations’ overall construction costs.
In this case, a utility had procured a large Chinese transformer. In the utility’s procurement specification, the utility identified a specific LTC vendor and model. As with many large equipment orders, the utility or their agent, will visit the manufacturer to assure the equipment is on schedule and meeting the specification requirements. While the utility’s agent was visiting the transformer manufacturer in China prior to shipping to the US, the agent checked for the serial number of the LTC to assure it met the procurement specification. However, the agent couldn’t find a serial number which should have been on all of the LTC vendor’s equipment. When questioned, the Chinese transformer personnel told the utility’s agent the LTC in this transformer was not from the vendor in the procurement specification (it was essentially a reengineered knock-off) but had similar functional capabilities to the vendor's equipment identified in the procurement specification. However, the transformer was urgently needed. Because of the additional time needed to order, replace, and reperform all testing with the LTC specified in the contract, the utility accepted the transformer with the knock-off LTC.
The issues of the compromised transformer and now the LTC have led to a myriad of issues that range from information sharing, supply chain, functional capability, cyber security, procurement, NERC supply chain requirements, training, governance, etc. In the July/August 2020 issue of IEEE Power and Energy, there is an article by PJM - “Toward Bulk Power Resilience”. The article includes sections on Spare Transformers and Cyberphysical Security. The Spare Transformer section addresses a Probabilistic Risk Assessment (PRA) model. The PRA model combines failure likelihood and congestion information to determine annual risk. However, a PRA cannot address malicious attacks. The Cyberphysical section addresses a comprehensive business impact analysis addressing hardware failures, software compromises, personnel availability, and disruptions to facilities or communications. The business impact analysis process includes ongoing penetration tests. However, penetration tests can’t help with systems with hardware backdoors installed.
I will discuss some of the issues involved in this potentially devastating problem recognizing the LTC case is another example where the Chinese have breached the ICS cyber security Maginot Line:
Supply Chain:
- The LTC is an integral part of the entire generation, transmission, and distribution supply chain. The transformer, whether it is a generator step-up transformer, a transmission transformer, or a large distribution transformer cannot work without the LTC. Depending on the scope of the supply chain assessment, it is unclear if the knock-off LTC would be identified. If the supply chain integrity program is based on the procurement specification, this situation would not have been identified. If the knock-off LTC would have had similar logos to the machine that was ordered, it may have required a functional test to determine whether this was what was ordered or a knock-off. This is why process sensor monitoring is needed to assure supply chain integrity.
Functional performance:
- The functional capabilities of the LTC identified in the procurement specification are well-known, with years of testing and in-service experience. The knock-off has no testing paperwork. As this wasn’t the LTC the utility ordered, it is unclear if the knock-off LTC can provide the required functional performance requirements. It is not safe to allow transformer operation with untested critical equipment.
Cyber security:
- The communication capabilities of the original LTC were specified and known. It is unknown if there was “additional” functionality in the knock-off that could enable unauthorized remote access or other capabilities. There is a need for sensor monitoring as it is unclear if the knock-off would only use the utility’s OT network and not some backdoor.
Regulatory:
- LTCs are out-of-scope for the NERC CIPS and NERC Supply Chain requirements. How can the LTC, a device so critical to reliability of the transmission grid, be out-of-scope for the NERC Supply Chain requirements yet routers and switches are in scope?
Information Sharing:
- The affected LTC vendor is arguably one of the most popular LTC vendors, and is used by many utilities domestically and internationally. The LTC vendor is also used by many transformer vendors. Consequently, this information about the knock-off LTC should be shared. But how and by whom?
Industry Transformer Pool
- The back doors in the WAPA transformer, the knock-off LTC, lack of Aurora protection, etc. means we may not be able to count on having back-up transformers. The aforementioned PRA model does not account for malicious attacks. What next?
Training
- LTCs are engineering, not network equipment. Many LTCs may not come with remote access, which can mean that the OT security organizations do not address this equipment. The knock-off was found by engineering and it is not clear this issue was communicated to the OT security organization. A similar issue occurred with WAPA, when the extra electronics was found by the site engineering organization. There is a need to have both engineering and OT security involvement and good communication between the organizations.
Governance
- As demonstrated by these cases, it is absolutely necessary that control system cyber security be a team sport with engineering being a lead participant.
Executive Order 13920 provides a great opportunity to address the cyber security of the control system and physical hardware. The Chinese (and Russians) are already in our grids. Without process sensor monitoring, backdoor connections may not be detected. When will the industry including regulators, vendors, industry organizations, equipment vendors adequately address these risks?
I will be discussing some of these issues in my July 15 Purdue Cerias presentation as well as the July 30th SURFA session.
Joe Weiss
Leaders relevant to this article: