Regulatory gaps drive systemic under-reporting and poor situational awareness

Malicious IT and OT network cyberattacks continue to occur, particularly ransomware. Malicious and unintentional control system cyber incidents also continue to occur in almost every sector – electric grid, power plants, water/wastewater, pipelines, manufacturing, transportation, etc.- yet they often are not identified as being cyber-related and therefore not reported. Requirements exist for identifying and reporting cyber incidents. Despite claims that information sharing is improved, it is evident that information sharing on control system cyber incidents is not working well in any sector.

Background 

Control system cyber impacts are visible – lights go out, pipes leak or break, trains crash, planes crash, etc. However, it is often not evident that cyber played a role. Many times, sophisticated cyber attackers will make a cyberattack look like an equipment malfunction. Because there are few cyber forensics at the control system field device level and little cyber security training for the control system engineers, these cyber incidents and attacks often go unidentified as having been cyber-related. Additionally, the lack of cyber security and authentication of control system field devices like process sensors make situational awareness suspect at best. This also means the incident response capability may not be initiated when the incident was not identified as being a cyber incident.  

DOE 2022 OE-417 results 

The electric industry has requirements for reporting electric disturbance events to DOE for the Electric Emergency Incident and Disturbance Report (Form OE-417).  Form OE-417 has been used to collect information furnished by the utilities on electric incidents and emergencies since 2000. Recently, DOE issued their OE-417 reports for 2022. The OE-417 reports are not explicitly for cyber incidents and so have to be analyzed further to address those incidents that are cyber-related. ). Many actual control system cyber incidents were not identified as being cyber incidents. Other control system cyber incidents that either did not cause a reliability impact or did not meet the reporting threshold also were not included in the OE-417 data.

According to the OE-417 data for 2022, there were 10 cyberattacks and 35 incidents of complete loss of monitoring or control capability at staffed Bulk Electric System control centers for 30 continuous minutes or more. (These were control system cyber incidents though they weren’t identified as such. Moreover, those incidents that didn’t meet the reporting threshold were not included meaning the 35 identified incidents are a very conservative number.) These 2022 incidents now make a total of more than 40 successful cyberattacks identified in the OE-417 reports since 2000 and more than 160 complete loss of monitoring or control  (this category was only established in 2018). The 2022 cyberattacks lasted in some cases more than 4 days. In one case, a cyberattack affected a multi-state utility’s multi-state grid operations, though unclear for how long. In another case, 80,000 customers were impacted by a cyberattack. There were 15 loss-of- view incidents that lasted more than 1 hour with 2 loss-of-view incidents resulting in 80 MW to almost 250 MW impacts. In one case, not only was there complete loss of monitoring or control capability at its staffed Bulk Electric System control center, there also was complete loss of Interpersonal Communication and Alternative Interpersonal Communication capability. 

Using sources in addition to the OE-417 reports, there have been more than 1,200 electric grid cyber incidents globally (this doesn’t include nuclear and non-nuclear power plants). There have been 7 U.S. cyber-related outages that affected at least 80,000 customers. Additionally, there have been grid cyber-related incidents in Europe, Asia, and South America that have affected millions of customers. 

Example Russian grid “attacks” on U.S. grids 

According to DHS, Russia installed the BlackEnergy 2 malware in US electric grids in 2014. This has given Russia visibility into the US grid, enabling it to gain awareness of grid topology and protections. Unfortunately, the NERC Critical Infrastructure Protection (CIP) cyber security requirements have no requirements to remove malware.  

In 2018, a utility discovered a compromised electronic access point connected to the Internet. The purpose of the internet-connected access point was to remotely access and operate substation capacitor banks to ensure the reliability of the system. Unauthorized personnel accessed the cyber asset for seven months before the utility became aware of the compromise. Additionally, the IP address and credentials were posted on a Russian-based media site, and the asset was subsequently infected with ransomware. The compromise was discovered after engineering staff could not remotely access the substation capacitor banks. This is not trivial as remotely accessing substation capacitor bank switches was a key aspect of the 2008 Florida outage that shut lights to millions of customers for eight hours.  

In 2022, a large US power plant experienced what appeared to be an Aurora event that damaged the generator shutting the plant down for an extended period. In this case, the grid was not affected as other generators made up the shortfall. The impacted power plant was in close physical proximity to a Russian-owned facility begging the question as to whether this was an accident or malicious. Now recall the Freeport LNG incident June 8, 2022 where the safety systems did not prevent the over pressure explosion. Again, were the Russians involved?   

Example Chinese grid attacks on U.S. grids 

DNI’s National Intelligence Council’s National Intelligence Estimate stated: “China is the world’s leading supplier of advanced grid components for ultra-high-voltage systems, such as transformers, circuit breakers, and inverters, which we assess creates cyber vulnerability risks.” 

China has delivered more than 200 large transmission-level transformers for use in the US grid since 2010. Additionally, US utilities use thousands of distribution-level Chinese-made transformers. Presidential Executive Order (EO) 13920 was issued May 1, 2020, because of hardware backdoors found in a large Chinese-made transformer. EO 13920 was meant to end the use of Chinese-made equipment in critical grid applications. Unfortunately, EO-13920 was suspended and US electric utilities are continuing to buy Chinese-made electric equipment. The focus on expanding the transformer pool has not addressed the cyber vulnerabilities in the transformers and supporting equipment.

China has also supplied counterfeit control system devices such as pressure and differential pressure transmitters which are major safety issues if they don’t work as expected. 

Self-inflicted wounds  

Much has been made of cyberattacks from Russia, China, Iran, and other malicious attackers. However, unintentional cyber incidents (or at least, not obvious intentional incidents) can cause similar impacts and continue to occur. Often, the reason is that the culture is broken. That is, the network security personnel do not understand the distinctive issues associated with the equipment they are supposed to be protecting, and the engineers do not participate in cyber security activities

In one case, an antivirus software engine on Energy Management System (EMS) production servers had a flaw that caused affected servers to become unresponsive. The flaw was not recognized in the patch testing. Two separate events over the span of two weekends led to a period of more than 30 minutes of complete loss of EMS functionality; this occurred again on the following Saturday for a period of more than 80 minutes. These performance degradation events removed the ability to control the impacted substations. NERC CIPs don’t address using patches appropriate to control systems. The International Society of Automation (ISA) has been developing a standard for patch management for control systems, but the NERC CIPs don’t reference this work. 

In another case, the utility IT personnel used new penetration testing software in their data center. Because it was successful, they used the same penetration testing software in multiple critical substations without testing the software with the relays in the substation. The IT organization also didn’t inform the substation personnel they were doing the testing. The penetration testing software was incompatible with the substation relays. As a result, the penetration testing software shutdown the relay communications to more than 400 high voltage relays. The impacts appeared to be from the Industroyer malware until the substation personnel realized it was caused by IT. This could have caused a major regional outage. 

In a third case, a utility had ordered a new distribution SCADA system. The purchase specification had identified the load shed algorithm to be manual, rather the automatic. However, the SCADA system was delivered with the load shed algorithm in automatic (apparently the factory and site acceptance testing did not identify this configuration error). As a result, when the SCADA system was delivered, the utility SCADA operator expecting the load shed algorithm to be in manual mode, selected load shed for testing. However, because the load shed algorithm was in automatic mode, the operator could not override the load shed signal and 400 MW of load was shed affecting 96,000 customers. Again, by luck, there was not a major regional outage. 

Recommendations 

The electric utilities cyber security program is based on meeting the NERC CIPs. The NERC CIPs were NOT developed to secure the grid. Rather, the CIPs were developed to have a documented set of policies, procedures, and training. Among other issues, the CIPs exclude electric distribution, process sensors and other field devices, pay minimal attention to “lower impact” systems even though they have directly contributed to major region-wide outages, and don’t address power flows. The NERC CIPs need to change to protect the grid, not just the IT and OT networks. There are woefully few engineers participating in cyber security programs. The self-inflicted wounds are testament to this gap. The government must reinstate EO-13920 or some similar version to prevent US utilities form buying Chinese-made products which may have software and hardware backdoors. Software and hardware bills of materials are useless when you can’t trust the Chinese vendors.