DARPA recently conducted a grid recovery project at Plum Island to determine the feasibility of grid recovery during and after malicious cyber attacks. (https://www.wired.com/story/black-start-power-grid-darpa-plum-island/). The Ukrainian utilities that were cyber attacked demonstrated that manual restoration was possible, particularly when they didn’t know if their SCADA networks were still compromised.
DARPA conducted a relatively small black start pilot exercise on Plum Island in June; the grid at that time was designed to be managed by a single utility running a diesel generator, known as Utility A, and a small cluster of substations. In the scenario laid out for the exercise, a massive cyberattack knocks some portion of the grid offline for weeks—long enough that residual power and substation batteries would all be depleted. Utility B's goal is to black start as quickly as possible, to deliver power to a customer that has been designated a critical asset.
The Plum Island test grid was designed to mimic the hodgepodge of technologies that coexist in real industrial control deployments. Vital systems like the grid can't be taken offline casually or overhauled easily, so equipment often remains in place for decades. Black start recovery, especially after a cyberattack, involves navigating, defending, and configuring generations of technologies. DARPA created sensors that can give accurate readings and situational awareness even after a hack has potentially skewed or degraded the reliability of existing monitoring equipment.
Industrial deployments are designed with process sensors (e.g., temperature, pressure, level, flow, voltage, current, etc.) to monitor and feed interlocks to assure that system safety is maintained. However, legacy process sensors have no cyber security, authentication, and adequate real time process sensor forensics is generally unavailable. An example is the temperature sensors that monitor turbine/generator safety systems to prevent the generators from operating in unstable or unsafe conditions. These process sensors are integral to the operation of the system and cannot be bypassed. If the temperature sensor is inoperable for any reason, it can prevent the turbine from restarting, whether in automatic or manual. The lack of generator availability can prevent grid restart from occurring.
The specially configured DARPA sensors rely on the insecure, unauthenticated process sensors in the equipment (e.g, diesel generators) that need to be restarted. In the case of the Plum Island test, it is not clear how the diesel generator process sensors were configured. What is clear is there are many generators and electric systems with process sensors that are connected and can be compromised.
These are not idle considerations. Process sensors have been hacked and the PLC and HMI’s have been unaware. In one case a process sensor was maliciously hacked and a turbine was unable to synchronize to the grid. In another case, a combustion turbine was unable to restart because a temperature sensor was bad (not malicious, but that’s irrelevant). The serial-to-Ethernet convertors that convert the process sensor analog signals for use in an IP network are a “two-way street” into the sensors as well as the networks and have been demonstrated to be cyber vulnerable. Changing sensor configurations such as span, range, or damping can affect reliability (prevent restart) or safety (remove equipment protection) with minimal forensics. Consequently, without on-line monitoring of the process sensors at the process sensor physics layer, it may not be possible to restart the grid.
It should be evident the lack of process sensor security is not just an electric grid issue. A petrochemical facility found instrument failure or improper ranging of scale were responsible for almost 50% of the nuisance alarms (or were they just nuisance?)
Last July, I gave a presentation at Defcon on the lack of process sensor cyber security. The presentation received a “like” on my Linked-In account from Iran. Several years ago, a Russian security research organization gave a presentation on compromising process sensor networks. The message is ignore legacy process sensors at your own risk.
For the first time, I will be speaking at S4 and it will be on the topic of sensors.
Joe Weiss
