Remote access for monitoring field equipment including turbines, pumps, valves, motors, transformers, electric substation equipment, port cranes, battery systems, etc. is important to maintaining equipment reliability and availability.
Modems (whether dial-up or cellular) that provide real-time equipment data to the internal organization, equipment vendors and system specialists have been used for more than 30 years. As an example, banks of modems are still used in many electric substations for access to satellite substations from a more central substation. These are slow (9600 bits/second or less) and operate continuously with no cybersecurity; that is, they’re given inherent trust. But because of the threat of hacking, remote access requires cybersecurity considerations.
Securing remote access is a very tough problem because remote access is often necessary for reliability and availability reasons. The basic premise is the equipment vendor or system integrator who has installed the modem is trusted, and the modem device is secure, as the modem becomes part of the equipment with 100% trust. Modems are two-way communication devices. That is, modems send data out for monitoring equipment health or other purposes, and they also receive data in the form, for example, of commands. Unfortunately, they can also send data out for industrial espionage and receive malicious commands for sabotage. Modems can send data in for applications such as firmware updates. Unfortunately, modems can also send spoofed sensor data to the equipment which treats sensor data as 100% trusted.
According to the Feb. 7, 2024 “Joint Guidance: Identifying and Mitigating Living off the Land Techniques” provided by the U.S., Canada, U.K., Australia and New Zealand, Living-Off-The-Land (LOTL) attacks are particularly effective because many organizations lack effective security and network management practices that support detection of malicious LOTL activity. This makes it difficult for network defenders to discern legitimate behavior from malicious behavior and conduct behavioral analytics, anomaly detection and proactive hunting. There is a general lack of conventional indicators of compromise (IOCs) associated with the activity, complicating network defenders’ efforts to identify, track and categorize malicious behavior. Whether exfiltrating data out or sending spoofed sensor data in, there is no need for an attacker to load malware, modify software or compromise the network because the modem is trusted. Moreover, network defenders may not understand what is being sent and the implications. Consequently, a compromised modem becomes the ultimate LOTL attack vector.
Modems are often supplied with the equipment as part of a maintenance contract. However, for equipment that didn’t have the modems originally installed by the equipment vendors or system integrators, modem installation is often done through the maintenance staff when equipment problems occur and need central engineering or off-site third-party expertise to diagnose the problem. As an example, I was at a large power plant meeting with the instrumentation & control systems lead. There was a Verizon truck at the plant. After leaving the plant, my plant contact and I met with corporate IT. When I mentioned the Verizon truck, the IT lead was stunned, as Verizon was not an approved IT/telecommunications supplier. This situation is normal. That’s not to say that there’s necessarily anything wrong with Verizon—someone at the plant hired them, after all. But it points up a potential problem: When modems are installed post-installation, IT and OT security personnel are generally unaware.
Unintentional
Monitoring combustion turbines are examples of what unintentionally can happen with remote access. The combustion turbine vendors require remote access to the combustion turbine which is owned by the combustion turbine supplier not the owner of the turbine. The remote access allows the vendor to monitor the health of the equipment and is required to maintain the warrantee on that “hundred-million-dollar investment.” Yet, in several cases, the remote links have shutdown combustion turbines. Several companies have tried to set up a policy that the link can only be actuated with plant personnel present and only at specific times. This is a good policy, but it is not always enforced.
Chinese equipment: backdoors to control systems
In 2021, the Director of National Intelligence (DNI’s) National Intelligence Council’s National Intelligence Estimate wrote: “China is the world’s leading supplier of advanced grid components for ultra-high-voltage systems, such as transformers, circuit breakers, and inverters, which we assess creates cyber vulnerability risks.”
Presidential Executive Order 13920 (since suspended) was issued because of the concern with the hardware backdoors in large electric transformers originally found at the Western Area Power Administration Ault substation in 2019. The Executive Order only addressed hardware issues including process sensors that have no cybersecurity, authentication or cyber forensics – network equipment was out-of-scope. The Chinese transformer manufacturer (JSHP) was publicly adamant there were no backdoors installed. The report on what was found in the Chinese transformer examined at the Sandia National Laboratory was classified as Top Secret, so I don’t have details. However, there is a possibility a modem was installed as that would be conventional industry practice. If so, the modem could be used to receive spoofed sensor signals to take control of the transformer.
According to a Congressional report, twelve cellular modems were discovered on crane equipment and in a server room at a U.S. port. Chinese crane manufacturer ZPMC rejected U.S. claims that communications equipment found on its port cranes were there for espionage purposes. ZPMC issued a statement, saying it “takes the U.S. concerns seriously,” but stressed that its cranes “do not pose a cybersecurity risk”. It added: “Cranes are designed, manufactured, transported, installed and commissioned and delivered after acceptance in strict accordance with international standards, applicable laws and regulations, and technical specifications determined by customers.” The statement neither confirmed nor denied the presence of the modems on the cranes, merely adding that ZPMC “continues to strictly abide by applicable laws”. ZPMC never explained the reason why they were there, never explained their use, did not provide the configuration settings, and downplayed the issue. Moreover, the port authorities had not specified the need for modems and did not know why the equipment had been installed.
A U.S. pharmaceutical facility had pharmaceutical equipment built by a Chinese company installed in a pharmaceutical manufacturing clean room. While doing maintenance on air handling equipment, a flow sensor and cellular modem were found that was not identified in the plant drawings. This led to a further detailed investigation where a “shadow” sensor network with a cellular modem was discovered. It was not clear if the shadow system had control capabilities or was just to exfiltrate data.
Duke Energy agreed under pressure from the US Congress to decommission energy storage batteries produced by Chinese battery giant CATL installed at Marine Corps Base Camp Lejeune in North Carolina over concerns that the batteries posed a security risk. The batteries and their inverters may have cyber vulnerabilities that could be used to compromise the electricity grid even though CATL said the battery system contained no cyber threats (just like JHSP and ZPMC). Duke Energy stated that the battery system had been designed with “security in mind” and that the batteries “were not connected in any way to Camp Lejeune’s network or other systems.” It would not be surprising for remote access to have been installed. What is surprising is Duke Energy’s blasé attitude.
Lack of government response
The ODNI Annual Threat Assessment of the U.S. Intelligence Community issued Feb. 5, 2024, did not explicitly address remote access capability to impact critical infrastructure. Rather it focused on Russia employing its civil and commercial remote-sensing satellites to supplement military-dedicated capabilities and has warned that other countries’ commercial infrastructure in outer space used for military purposes can become a legitimate target such as the February 2022 Russian attack on Viasat’s satellite network that erased modem capability.
The Strategy for Cyber-Physical Resilience: Fortifying Our Critical Infrastructure for a Digital World Executive Office of the President President’s Council of Advisors on Science and Technology issued February 2024 did not address remote access which would be critical to maintain cyber-physical resilience.
The DOE and NARUC February 2024 report Cybersecurity Baselines for Electric Distribution Systems and DER addressed remote access. That is, security is necessary to make sure an unauthorized user could not get to the remote access. Consequently, the report addressed Phishing-Resistant Multifactor Authentication (MFA) Implement MFA for remote access to assets using the strongest available method for that asset. But that doesn’t apply to a “trusted” supplier that would have appropriate MFA credentials.
Recommendations
Treating modems used for remote monitoring and control as inherently trusted is a risky practice. Both technical and programmatic requirements need to be modified or added. As remote access is used for operations and maintenance needs, the engineering organizations need to be involved. All modems need to be identified with a name responsible for the remote access device. Strict instructions need to be applied as to when the remote access can be used and by whom. Communications should be monitored to assure that confidential information is not being exfiltrated. Where possible, communications should also be one-way, data out – no data in though it may be problematic for power system relays and some industrial relays. Remote access security tools should be employed. Never install a device, or accept a device, that you cannot directly monitor, know the configuration of, know the endpoints for the destination of the data and are installed in places that have limited to no physical access (hard-wired in places that restrict direct access by the network administrators or technicians). Modify procurement requirements to preclude use of Chinese equipment. Modify the cybersecurity program to include detailed site acceptance testing to identify any modems included.
Summary
Remote access to control systems is necessary for equipment reliability and availability. However, remote access can be a double-edged sword providing needed reliability improvement but also a potential vehicle for Living-off-the-Land attacks. Cybersecurity technologies exist to secure remote access from external intruders. Compromised remote access have been found in different industries from different vendors with the common threat that the equipment was Chinese. Cybersecurity programs are not adequately addressing the “trusted” insider. In many cases, this is the Chinese equipment vendors supplying the modems.
