This is arguably one of my most important blogs as it directly relates to how the Russian’s got a “two-fer” in the SolarWinds hack – compromise of the IT infrastructure and direct CONTROL of building control system devices. The Russians also got indirect control of “industrial” control systems via the IT network backdoors.
A highly sophisticated Russian Intelligence group has compromised the SolarWinds Orion platform. The platform’s broad base of users has been estimated at up to 18,000 customers and includes an unknown but vast number of sites. FireEye, which was affected by SolarWinds, issued a blog on SolarWinds dated December 13, 2020, “Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor”. The blog states: “They gained access to victims via trojanized updates to Solar Wind’s Orion IT monitoring and management software. This campaign may have begun as early as Spring 2020 and is currently ongoing. Post compromise activity following this supply chain compromise has included lateral movement and data theft.”
I have been attending several SolarWinds webinars. The FireEye blog, the DHS CISA (https://us-cert.cisa.gov/ncas/alerts/aa20-352a) and the SolarWinds Advisory (https://www.solarwinds.com/securityadvisory), and the webinars have all focused on the IT networks, network visibility, and data exfiltration/compromise. However, SolarWinds is used to manage all types of Simple Network Management Protocol (SNMP) devices and these include not just IT equipment like servers and switches but, also Industrial Control Systems (ICS) like power and cooling systems. Specifically, SNMP management systems are used to monitor (using the SNMP "get" command") and control (using the SNMP "set" command) any SNMP device. However, I have not heard any discussions that SNMP management platforms like Orion also monitor and control critical power and cooling systems in mission-critical facilities even though SNMP has little or no viable security. As noted in https://www.controlglobal.com/blogs/unfettered/data-centers-have-been-damaged-and-they-are-not-being-adequately-cyber-secured/, data center equipment have been compromised using SNMP vulnerabilities.
Most power and cooling systems in data centers, laboratories, telecom systems, and network closets include SNMP communication cards or chips for the expressed purpose of allowing them to be monitored and remotely controlled. These power and cooling systems include:
- Switchgear
- Power Distribution Units
- Rack Power Distribution Units
- Uninterruptible Power Supplies
- Rack Uninterruptible Power Supplies
- Computer Room Air Conditioners and Air Handlers (CRAC and CRAH units)
- Temperature and Humidity Sensors
- Rack Environmental Monitoring Systems
In mission-critical facilities, it is common for multiple management platforms to have monitoring and control access to SNMP-based systems. For example, the Network Management System (NMS) like Orion typically tracks device information at the rack-level, including the servers, switches, Rack PDU's and, when present, Rack UPS, Rack Cooling, and Rack Monitoring systems. At the same time, Building Management Systems (BMS) track the central cooling units and rack cooling units while the Data Center Infrastructure Management (DCIM) system is also tracking Rack PDUs, Main PDUs, Main UPSs, and Switchgear. This gap in building risk assessments was highlighted by the blog - https://www.controlglobal.com/blogs/unfettered/lack-of-iot-hvac-control-system-cyber-security-and-potential-real-world-impacts/
Even the most recent SNMP version, version 3, is now over 20 years old and has long been shown to be vulnerable (though not as vulnerable as versions 1 and 2). This peer-reviewed and well-researched report from the Georgia Institute of Technology shows just how easy it is to compromise SNMP devices and cause physical damage.
Devices that are using SNMP are insecure and can easily be compromised. The Russians used SNMP communication cards as attack vectors in their 2015 attack on the Ukrainian power grid that left hundreds of thousands without power. The Russians used targeted the UPS in the control center as the starting point for their attack. In short, they placed code on the UPS that made it shut down at precisely the moment when they also started a Denial of Service (DoS) attack on the telecom switch (just as shown above in the consequences of SNMP attacks as outlined by the Georgia Tech paper). Simultaneously, they opened the main breaker that delivers power to the Ukrainian power grid serving much of the country.
It has long been speculated that the Russians have been using the Ukraine as the "test laboratory" for more extensive cyberattacks they intend to use against other countries. Unfortunately, we can now say that they have succeeded in this SolarWinds Orion attack. While almost all are searching for malicious code on servers and managed switches, the reality is that code has likely been placed on power and cooling systems' SNMP cards throughout mission-critical enterprises around the globe. SNMP malware is extremely difficult to detect, and there are only a handful of people who have been involved in finding and managing this type of malware. Bob Hunter is one such person, and he and I have collaborated and documented SNMP malware. Bob and I co-authored the “Cybersecurity in Data Centers” Chapter in the upcoming 2020 Data Center Handbook by Wiley & Sons, and he provided input into this blog. I would encourage you to read his early post on the SolarWinds hack and to look at his company's secure SNMP systems.
The Russians spent significant efforts to deliver the BlackEnergy2 malware that is still in the US grids. The Russians modified it to create BlackEnergy3 which was used in the Ukrainian power grid attacks where the starting point was hacking the UPSs which use SNMP. Now they spent a significant amount of effort to develop SolarWinds which enables direct access to the SNMP devices in more than 18,000 customers. These 18,000 customers have critical buildings (e.g., data centers, control centers, laboratories, manufacturing buildings, etc.) with SNMP devices. Now reread https://www.controlglobal.com/blogs/unfettered/lack-of-iot-hvac-control-system-cyber-security-and-potential-real-world-impacts/ to get a sense of what may be awaiting these targeted organizations.
This attack demonstrates the need for the paradigm shift for control system cyber security by having an independent view of the control system devices not connected to any IP network.
At what point will control systems be adequately addressed?
Joe Weiss
