Control system cyber incidents can be either unintentional or malicious. Depending on the sophistication of an attacker, it might not be possible to tell the difference. I have documented almost 100 control system water/wastewater cyber incidents. These cases are geographically dispersed across the U.S. as well as internationally. Not all cases could be identified as cyber-related (for example, in one case the intruders entered the SCADA system via remote access and deleted all computer system logs and alarm history). Many incidents were unintentional. However, the impacts could be devastating. An example was a water utility that inadvertently pumped water from a Superfund contaminated well site into a drinking water system.
There have been suggestions the Oldsmar cyberattack occurred because of the lack of enforceable cyber security regulations in the water industry. As stated in the Wall Street Journal’s article by Rebecca Smith on February 12, 2021, “U.S. Water Supply Has Few Protections Against Hacking”:
“The hacking incident—occurring after a security review—has thrown into stark relief a vulnerability of the more than 50,000 community water systems that supply most Americans with their drinking water: they don’t have to meet any national standard for cybersecurity. That is in contrast to electric utilities, which have had to meet increasingly stringent rules since 2008 for the physical and cybersecurity of key assets and, more recently, for parts of their supply chains. Rules for the electric industry are reinforced by monetary penalties for violations.”
However, Smith’s conclusions about the electric industry are not completely correct. The remote access issue should be addressed by the electric industry cyber security standards assuming the facility is large and critical enough to be covered by the North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection (CIP) cyber security standards – most are not! The Florida event attempting to inject too much sodium hydroxide is a control system issue that affects power generation not power delivery (electric transmission and distribution). However, the control system settings in power generation that could cause these issues are out-of-scope for electric industry cyber security requirements regardless of the size or criticality of the plant. Because of these regulatory gaps in electric industry regulatory standards, the electric industry is at best in a marginally better position to have prevented the Oldsmar-type cyberattack. Consequently, the water industry emulating the electric industry’s cyber security requirements may not be as beneficial as Smith implies.
The February 5, 2021 Oldsmar, Florida hack
The Oldsmar water treatment facility experienced a malicious hack of the remote access system (TeamViewer) to the SCADA system. The hacker sought to increase sodium hydroxide (lye) concentrations in the water treatment system to an unhealthy level. The hack was detected by the operator and damage was prevented. When the Oldsmar cyberattack was made public by the Sheriff, many people reacted as if it was the first hack of a water system and the hack was just a remote access problem. However, this was not the first case of compromising remote access to a water facility SCADA system nor the first case where the sodium hydroxide system was impacted by a control system cyber incident. A similar event occurred almost 14 years ago.
2007 Massachusetts sodium hydroxide incident
In 2007, a wastewater utility in Massachusetts completed maintenance of the chemical feed pump (sodium hydroxide) system (just like Oldsmar). In this case, the controller was inadvertently left in the manual rather than automatic mode. Consequently, the system continued to add sodium hydroxide rather than automatically limit the concentration. As a result, the pH high/low alarm sounded to indicate elevated pH levels in the water. However, the alarm was not connected to the auto-dialer, so the alarm only sounded within the building without notifying the appropriate people outside the building. In this case, even if the auto-dialer worked, the phone line was dead. Eventually the high pH alarm was acknowledged resulting in the utility declaring a state of emergency. The high pH conditions resulted in the water regulator ordering a complete ban on water use for 2 days. The incident resulted in more than 100 people being hospitalized. The two senior operators were fired. This incident can explain why the State of Massachusetts was so quick (and the only state) to issue recommendations following the Oldsmar cyberattack. It should be obvious this incident could have been maliciously caused like what happened at Oldsmar.
2011 SCADA remote access breach
In 2011, a water utility suffered a security breach of their remote terminal server that enabled remote access to SCADA. The remote terminal server appeared to be running a computer-generated hacking program that was trying to gain access. The utility also noted two user accounts on the system they did not put there. The utility reported this incident to their representative on the Joint Terrorism Task Force. This incident was originally identified on Linked-in (subsequently deleted). In a phone call, the SCADA supervisor acknowledged the event had occurred and was looking for help. However, subsequent to the DHS flyaway team arriving at the utility, the SCADA supervisor mysteriously disappeared and all communication ended.
2011 Illinois water hack
In 2011, the Illinois Fusion Center issued a report titled “Public Water District Cyber Intrusion”. Like the above-mentioned case, there was no disclosure of this event to the water industry by either DHS or the Water ISAC until after I issued my blog. Specifically, the Fusion report stated, "Over a period of 2-3 months, minor glitches were observed in remote access to the water district's SCADA system. The SCADA system powered on and off resulting in the burnout of a water pump. The glitches were observed months after the system integrator returned from a vacation in Russia. The water district General Manager initially acknowledged the cyberattack until after the DHS flyaway team arrived at which point this case was no longer considered to be a cyberattack.
2016 “Kemuri” incident
A corollary to the Illinois water hack which has been dismissed by industry is the Kemuri water cyberattack identified by Verizon Security Solutions in 2016. There were two issues with Verizon report – Kemuri is a fictitious name and Verizon is not a company with water systems engineering expertise. Yet, the Kemuri case has been accepted by many in the industry as real. According to Verizon, hackers infiltrated a water utility’s control system and changed the levels of chemicals being used to treat tap water (similar to Oldsmar). According to Verizon, a "hacktivist" group with ties to Syria compromised Kemuri Water Company’s computers after exploiting unpatched web vulnerabilities in its Internet-facing customer payment portal.
Questions for water utility cyber security
The previous cases represented the key aspects involved in the Oldsmar cyberattack – remote access security and safety of sodium hydroxide systems. The guidance about securing remote access is important and needs to be followed but that is not sufficient. Consequently, these cases raise the following questions:
- Why do these incidents continue to occur if information sharing is available?
- Why are real cases disputed and hypothetical cases accepted?
- Why aren’t there appropriate monitoring and testing regulations to keep these incidents from recurring?
The same questions could be asked of any other industry as they have experienced similar situations.
A need for appropriate training and reporting
Government and industry need to work together to identify and mitigate control system cyber incidents in water and other critical infrastructures. Because many water utilities depend on DHS and the Water ISAC for cyber information (as do other industries with their ISACs), the lack of sharing of control system cyber incidents can significantly impact industry preparation - https://www.controlglobal.com/blogs/unfettered/information-sharing-on-control-system-cyber-incidents-isnt-working-the-chinese-transformer-case/. This goes beyond DHS and vendor efforts to identify network vulnerabilities. Besides my non-public database, there are few other efforts to identify control system cyber incidents including infracritical’s SCIDMARK. Control system cyber security training and mitigation technologies should be based on real cases or extrapolation from real cases. However, there has been little move to incorporate this information which can be done on an unattributed basis.
A need for appropriate regulation
The Oldsmar cyberattack was not just about inadequate remote access but also about the need to include engineering considerations as shown in the 2007 sodium hydroxide case. Given the number of actual water control system cyber incidents to date along with the feared hacker focus on water facilities, water industry cyber security regulation is needed to prevent cyberattacks that can affect the health and safety of the public. The water plants may be small facilities, but the impacts can be very large. Consequently, the regulation needs to be based on comprehensive risk assessments and include appropriate training, reporting, and engineering system assessments as well as network security. This approach can be used by all critical infrastructures.
Joe Weiss
