Michigan-based Waterford Township's Dept. of Public Works is upgrading its water/wastewater controls and SCADA system for 63 sewer lift station and 11 drinking water plants with MQTT publish-subscribe protocol running on Opto 22's groov EPIC controllers. MQTT sends data on a change-of-state basis, which uses less bandwidth, achieves sub-second latency, and provides more useful information more quickly. The utility adopted a cloud-based AWS server, and is securing it and the new SCADA system with passwords, firewalls with whitelisted IP addresses, security certificates, TLS data encryption, and multi-location server backup. Source: Waterford Township and Opto 22
Similar to training and retraining staff, one of the best ways to instill cybersecurity protections is to make them simple, so users will be more likely to get in the habit of using them, maintain them over the long-term, and seek updates when available. Until recently, many cybersecurity solutions were bolted onto existing processes, which was often technically complex due to old, dedicated and rigid network infrastructures, and so less likely to be implemented and maintained. Fortunately, adopting publish-subscribe protocols like MQTT and others not only simplifies networking, it can also streamline cybersecurity.
For example, Waterford Township's Dept. of Public Works (DPW) began upgrading its 1990s-era SCADA system, field controllers, radios and other components in 2017, but decided to reevaluate its entire control strategy. Located in the center of Oakland County, Mich., the township has 34 lakes, and DPW serves more than 73,000 residents by operating 360 miles of water mains, 355 miles of sanitary sewers, nine production wells, three storage tanks, 11 treatment plants and 63 sewer lift stations. As a longtime innovator in water/wastewater management, DPW previously integrated core applications, such as its SCADA system, geographic information system (GIS), asset management system (AMS) and enterprise content management (ECM), which already share data with each other to enable seamless operations.
Skip central polling
"Polling times weren't great, and it was taking three or four minutes to cycle the whole township system," says Frank Fisher, engineering superintendent at DPW. "We'd already modernized our field controls and update our in-office SCADA/HMI and backend, but we evaluated a hardware upgrade in 2018, and found that MQTT protocol running on a groov EPIC controller from Opto 22 would allow faster information transfers with no central polling required, instead providing data on change. So, we added groov EPICs to a pilot project in 2019, consisting of three of our sewer lift stations with help from system integrator Perceptive Controls in Plainwell, Mich. We just finished adding the last of our 63 stations two months ago, and also upgraded six of our 11 drinking water treatment plants. Three more plants are converting now, and we'll finish them and the rest by 2022."
This article is one in the 2021 cybersecurity update multi-part series.
View the rest of the series here.
Instead of polling every two or three minutes, even when nothing is happening, Fisher reports that MQTT monitors the water/wastewater system's overall heartbeat every few seconds, but only sends data when process values change. This gets more useful data across at sub-second latency and reduced bandwidth, which means alerts and alarms aren't missed, and diagnostics are improved. DPW also adopted an Amazon Web Services (AWS) server for its new system. This cloud-computing service hosts Ignition software from Inductive Automation as the utility's MQTT broker and SCADA.
To protect DPW's new cloud-based SCADA system and its additional, potentially vulnerable connections, Fisher reports that DPW limits electronic access to its lift stations and other equipment. These layers of protection include login passwords for its virtual server and controllers, and continues with firewalls on the AWS side and cellular modems with predefined IP addresses for which communications are permitted.
Set up certificates
"We also use security certificates to encrypt data before sending it, so even if traffic is intercepted, it can't be read," explains Fisher. "Previously, our radios transmitted data openly, but not much cybersecurity was required because we had lockouts on the buildings and still have them. They only talked to our onsite server and only certain PCs could access them."
Each groov EPIC also uses a certificate authority (CA)-signed transport layer security (TLS) certificate to establish a secure broker connection. Also, with MQTT’s device-originating connections, groov EPIC's device firewall in each lift station can be closed to outside connections. Based on this foundation, Fisher reports DPW made other infrastructure changes to enable management, security and resilience. For instance, its Ignition SCADA/MQTT server is hosted in a data center in Ohio, but if any issues arise, in 30 minutes, Fisher or his colleagues can have their entire system up and running on a snapshot of the same server hosted by a data center in Oregon. In the future, DPW will likely set up full server redundancy that also avoids potential single-point-of-failure problems.
"Cybersecurity begins with each user researching and deciding internally what they need," adds Fisher. "Plus, as we use the Industrial Internet of Things (IIoT) to communicate with edge devices we haven't talked to before, such as sensors and flowmeters, cybersecurity will be even more important. MQTT opens all of this, but it's also like a web browser, so users have to turn on its security functions, such as requiring security certificates and encryption. This is similar to turning hypertext transfer protocol (HTTP) into hypertext transfer protocol secure (HTTPS)."
Creating secure zones
Josh Eastburn, technical marketing director at Opto 22, adds that, "A lot more remote connections have been added lately, but all the preaching about adding cybersecurity to existing systems doesn't address the fact that they weren't designed for it. So the question for users is how to adopt cybersecurity without breaking the bank. Do they 'rip and replace' or 'layer and prayer'?"
Eastburn reports that Opto 22 tries to offer a middle way, splitting the difference between replacing existing devices and software and adding broad security layers that complicate management and connectivity. "Our groov EPIC controller provides a defense layer that creates a secure zone around critical field assets. It lets us segment IP networks with a firewall, defining what’s allowed to pass through, while still allowing them to exchange data, even at the device level."
In general, Eastburn agrees that securing field devices and production processes begins with segmenting networks and turning on user and device authentication, but it can't stop there. "Users have to encrypt connections to external zones, like business networks, cloud services, MQTT brokers and remote VPN servers by using TLS-encrypted certificates for each participant," explains Eastburn. "They also need parity in the applications trying to reach them from the other side. This means MQTT used in the field, a SCADA package, or applications at the IT level need to use the same connection security practices."
