Many people are talking about the need to regulate pipelines for cyber security. I have documented more than 50 gasoline and natural gas pipeline control system cyber incidents including several that killed people and caused extensive property damage. The 1999 Bellingham, WA Olympic Pipeline gasoline pipeline rupture is documented in my book - Protecting Industrial Control Systems from Electronic Threats. This was a control system cyber incident that killed three people, three people went to jail, and led directly to the bankruptcy of the Olympic Pipeline Company. The 2010 Pacific Gas & Electric (PG&E) San Bruno natural gas pipeline rupture killed 8 people, destroyed a neighborhood, and resulted in PG&E becoming a convicted felon. As a result of the San Bruno natural gas pipeline rupture, the federal government required natural gas pipeline operators to evaluate risks in their systems and take action including the installation of remote, automated shut-off valves – making the natural gas pipeline industry cyber vulnerable.
As a result of having done the analysis of the Olympic Pipeline rupture, I expected the Colonial Pipeline Company hack to have been an OT incident affecting the SCADA system and potentially causing pipe leaks or pipe ruptures. Consequently, I had planned to respond about the control system cyber security implications of the Colonial Pipeline cyberattack. However, that does not appear to be the issue in this case.
There are three issues in play – ransomware, IT/OT convergence, and pipeline cyber security. The scary fact that almost any person in an organization that clicks on a “poison” attachment could cause a problem of the magnitude of what happened with Colonial Pipelines should give everyone pause for thought. The more I dug into the problem, the more complicated it became. Consequently, this blog is really about what is not known including:
- Did the attack come from IT or OT?
- What data critical for pipeline operations were held hostage?
- If there was OT network monitoring, was the attack visible to that monitoring?
- Were there any OT or safety systems directly accessed or compromised by the ransomware?
- Was the organization responsible for the normal operation of the pipeline system aware of the potential cyber implications of an IT compromise?
- What guidance will CISA provide about the potential unintended threats brought about by IT/OT convergence?
Ransomware is an epidemic affecting organizations world-wide and is getting worse. Organizations that have been impacted include retail, banking, schools, healthcare, industrials and manufacturing, and others. Recall WannaCry/NotPetya and other attacks caused billions of dollars of impacts related to unplanned and unscheduled downtime, though very little, if any, direct physical damage to equipment. The typical impacts on manufacturing and stated by industrial facilities has been either due to loss of IT data or suspension of operations due to an “abundance of caution”. Demonstrating credit rating agencies’ concerns about the Colonial Pipelines cyberattack, Moody’s Investor Services issued a report “Colonial's suspension of pipeline operations shows sector's vulnerability to cyber risk.”
The Colonial Pipeline cyberattack was an IT ransomware attack meant to steal data and lock up IT systems not cause physical damage. The Darkside Ransomware Group's malware is IT ransomware and not custom built for ICS attacks. As the Darkside Ransomware Group stated on May 10th commenting on the consequences of their attack: "We are apolitical, we do not participate in geopolitics, do not need to tie us with a defined government and look for other our motives. Our goal is to make money, and not creating problems for society. From today we introduce moderation and check each company that our partners want to encrypt to avoid social consequences in the future." I do not believe this should be viewed as a new international norm that will apply to hacker gangs or adversarial nation-states as it would be naïve to take criminals at their word.
Learn more about the Colonial Pipeline hack in this Quick-Response Report.
The attackers stole nearly 100GB of data from Colonial Pipeline before locking some of its computers and servers and demanding a ransom. This description could be from any of the myriad ransomware attacks across all sectors. However, this is where it starts to become more complicated. Apparently unbeknownst to the hackers, Colonial Pipeline has comingled IT and OT (IT/OT convergence) as part of the operation of the pipeline systems making this a national societal threat. As Gary Rathwell stated: “It is also worth noting that operation of a product pipeline is remarkably complex. Tracking of where various products are that are often separated by only “markers” (soft pigs or various injected dyes) is central to billing and tracking ownership of millions of dollars of material. There is also the issue that products are scheduled in increasing API (density/viscosity) so you can’t just start operating again if you “lose the plot”. This is probably also a reason for the slow restart of their pipeline network.” Because the pipeline carries multiple products, it becomes both a financial and even safety issue if you don’t know what product is in the pipeline. To date, the cyberattack has not appeared to have affected the control systems or OT networks indicating the OT networks either were isolated from the IT networks or they were not the target of the attack.
From Kim Zetter's blog, another expert brought up that a reason that Colonial Pipelines might still be keeping the pipelines offline is that “something they need for [restarting] the pipeline is ransomed.” This could be the automated ticketing system for billing customers, which is on the corporate IT network that apparently was hit with the ransomware. If that system were locked, Colonial Pipelines couldn’t invoice customers automatically. Colonial Pipeline’s OT network controls the flow of oil product from the pipeline to distributors, then passes information to the ticketing system — located on the IT network — about how much each distributor received so the ticketing system can invoice them. If that system is locked and the pipeline is still flowing, Colonial Pipeline would have to manually collect information about how much fuel is flowing to each customer, then manually process invoices. If Colonial Pipeline didn’t already have a plan for doing this manually, it may keep the pipelines down until it can determine an efficient way to invoice customers this way or until it can restore the automated ticketing system.”
In either case, the fundamental issue is that data necessary for pipeline (facility) operations should not be resident on an IT network except as a backup. That is, any attack on the IT network should never be able to affect pipeline (facility) operations or safety. In the Colonial Pipelines case, data necessary for the operation of the pipelines were on the IT network and held hostage necessitating the shutdown of the pipeline system. As ransomware is the withholding of data not sending malicious data, a ransomware attack of an IT network would not be expected to directly be found by OT network monitoring. The comingling of IT & OT networks brought about by Digital Transformation/Industry4.0 initiatives have expanded an IT footprint into OT environments. OT is leveraging the same and or similar technologies as traditional IT devices (e.g., Window Operating Systems, Cisco Switches, etc.) to operate and communicate OT applications such as SCADA. An attack on Windows operating systems sees no barriers between IT & OT connected devices. Additionally, OT environments are 10 to 15 times the size of IT’s footprint, just in the sheer number of connected devices.
Until recently, I was not aware that ransomware could cause physical damage to facility equipment or people. Unfortunately, this has happened and cannot be excluded. In this case which was not associated with the Colonial Pipeline attack, a member of a commercial building’s facility’s staff was spear phished to install ransomware on the IT business network similar to the Colonial Pipeline case. Unfortunately, cyber security was not a priority at this facility and among other concerns:
- the business network that was targeted with the ransomware was directly connected to the building control system OT network,
- the engineer’s workstation was left on,
- ....
As a result, the attackers were able to pivot from the business IT network to access the building control system network where they able to access critical control system equipment. The impacts included taking control of the Variable Frequency Drives (VFDs) controlling water pumps and operating the VFDs beyond critical operating speeds. With the VFD safety features and alarms disabled, pumps, VFDs, and controllers were damaged. Everything that occurred here could occur in gasoline or natural gas pipelines causing physical damage with minimal control system cyber forensics.
Specific to SCADA/control system issues, process sensors are used throughout the pipeline industry for monitoring, control, and safety issues as well as for pipeline leak detection. Process sensors and sensor networks (not IP) often are the least understood part of control system cyber security (and therefore generally ignored), yet they can have some of the most significant impacts including contributing to refinery, pipeline, and tank farm explosions. Process sensors are assumed to be secure, authenticated, and correct. Yet they are not. As process sensors are generally not native to IP networks, they can provide an independent view of the process and not be affected by ransomware. Russia, China, and Iran are aware of the cyber security gaps in these devices and in some cases appear to be currently conducting reconnaissance or even staging future attacks on such systems.
Conclusion
Ransomware is an epidemic that can be expected to continue. The issues that occurred with the Colonial Pipeline ransomware attack are not unique to pipelines or any other critical infrastructure as the IT/OT convergence is moving critical operational data to IT without the proper controls or visibility. With the hacking of IP networks, there is a need to detect operational changes independent of the OT network which can be accomplished by monitoring the physics of the process sensors. Control system cyber security and the appropriate integration with IT security needs to be stepped up to prevent ransomware IT hacks from causing physical damage and causing significant societal upheavals.
Joe Weiss
