Attackers wanting to cause damage will often use novel approaches that defeat exiting monitoring approaches. This blog addresses several hardware attack vectors meant to cause physical damage that are not being adequately addressed. Moreover, these approaches put very critical facilities at risk.
Cyber threats to data centers include data issues: compromise, exfiltration, and denial-of-service. Control system cyber threats to data centers have focused on the Internet-connected building control systems. However, there are other control system cyber threats to data centers that have not been addressed and have actually caused data center damage. Control system network vulnerabilities include the use of standardized cyber vulnerable communications protocols such as Modbus/TCP, BACnet and SNMP. These protocols have been demonstrated to be vulnerable to cyberattacks and, in the case of Modbus, there are simply no security features built into the protocol at all. Hardware vulnerabilities include the Aurora vulnerability and Uninterruptible Power Supplies (UPSs).
Aurora vulnerabilities occur when electric substation breakers are opened and then reclosed out-of-phase with the grid. This will generate large torques and current spikes that will damage or destroy and Alternating Current (AC) equipment connected to those breakers (see https://video.search.yahoo.com/search/video?fr=mcafee&p=cnn+auroara+generator+test#id=2&vid=dd006b5cfc280fed537e950070a492e0&action=click ). The Aurora demonstration proved there could be physical damage from an attack though the operators were blind because the attack was not see from the SCADA system. An actual Aurora event affected a data center when the data center experienced multiple Aurora events over a multi-day span. The events originated from the utility which was outside the facility’s control. The Aurora events damaged chiller motors with one of the motors out of operation for weeks. The controller logs showed no breaker operation though the mechanical counter showed breaker operation. (This is similar to what occurred with the March 2007 INL test.). Aurora vulnerabilities originate from outside the data center. Data centers have assumed that the electric utility substations feeding the data centers have addressed Aurora. However, this is generally not true.
UPS generally serve two functions. UPSs smooth the voltage from the backup generators so the servers are only fed the design voltage, rather than the fluctuating voltages and frequency produced by a local generator as the load varies. UPSs also supply interim power when power is lost from “house loads” until backup generators/batteries kick in. UPSs are remotely accessible yet, are assumed to be secure and available (Bob Hunter from AlphaGuardian provided input on UPS SNMP cards for this blog). Compromising the UPSs can directly lead to data center equipment damage. SNMP management cards are an integral part of most every company’s power management system. SNMP cards were developed about 25 years ago with the advent of SNMP version 1. The majority of all SNMP cards are still running version 1, which has NO security, or version 2, which has minimal security. Even cards that support version 3 can be compromised by a competent hacker - https://www.usenix.org/system/files/conference/woot12/woot12-final14.pdf.
In the December 2015 Ukrainian cyber attack, the attackers discovered a network connected to a UPS and reconfigured it so when the attacker caused a power outage, it was followed by an event that would also impact the power in the energy company’s buildings or data centers/closets. The outage left nearly 250,000 people without power and caused enormous suffering to many residents within a wide area.
May 27, 2017, British Airways’ reported that their Boadicea House data center experienced a major power outage due to an electrical grid power surge. However, National Grid confirmed there were no problems with its transmission network and Scottish and Southern Electricity Networks, the local electricity distribution network operator, also recorded no problems on the local distribution side. Further, no other companies near the area of the British Airways data center reported any type of power anomaly. Consequently, any change in power had to occur from within the data center. According to the head of Group IT at BA's owner International Airlines Group (https://www.theregister.co.uk/2017/06/02/british_airways_data_centre_configuration/ ), a subsequent investigation found that a UPS was over-ridden resulting in a hard power shutdown. While the UPS is supposed to act as the first line of defense in an actual power event, it can also be used at the first line of attack in a cyber/physical attack. In this case, all UPS-supported power to servers and network equipment in the data center was shut down. This resulted in the total immediate loss of power to the facility, bypassing the backup generators and batteries. This meant that the controlled contingency migration to other facilities could not be applied. After a few minutes of this shutdown of power, the UPS was just as mysteriously turned back on in an unplanned and uncontrolled fashion. The result was both the battery supply and the generator supply being connected in series to the power bus feeding the racks. That resulted in the data center’s servers being fed 480v instead of 240v, causing physical damage to the servers and significantly exacerbated the problem.
BA’s Boadicea House data center utilized Socomec’s Smart Powerport UPS which, like other UPSs, utilizes SNMP interface cards. Specifically, SNMP interface cards allow the following:
- Shut down a UPS immediately
- Schedule a shutdown and restart of the UPS at a specifically scheduled time
- Turn off power to selected small UPS outlets or large UPS phases
- Drain or disconnect the backup batteries
Because of the lack of control system cyber forensics, it is not possible to absolutely determine if the UPS compromise was a cyber event. However, all aspects of the BA data center event could have been caused by hacking the SNMP interface cards.
UPS issues are not just data center vulnerabilities. The 2010 PG&E natural gas pipeline rupture in San Bruno, CA occurred when PG&E scheduled a SCADA UPS replacement. The UPS replacement resulted in a SCADA shutdown (low voltage). On SCADA low voltage, PG&E’s control system logic opened the control valves leading to an over-pressurization event and a rupture of a weak pipe in San Bruno destroying a neighborhood.
All network-connected power systems, not just UPSs, can be cyber vulnerable. Other power systems that are cyber vulnerable because of their reliance on Modbus/TCP and SNMP communications include Power Distribution Units (PDU’s), Smart Breakers, Automatic Transfer Switches, generator systems and many others.
The common thread between Aurora and the UPS attacks are that systems that were designed to protect mission critical systems have been co-opted to be used as attack vectors against the very systems they were meant to protect. UPS and generator systems are very expensive pieces of power infrastructure that are used to protect critical system/facilities but they have weak links with their communications cards, which typically cost less than $1000. In order to insure that a UPS, generator or other critical power system cannot be hijacked and used as a weapon, it is critical to understand the cyber threats to this equipment and employ appropriate cyber protection to both monitor and protect these systems.
Joe Weiss
