Figure 1: East Cherry Creek Valley Water and Sanitation District in Colorado is upgrading the PLCs, RTUs, radios and SCADA system for more than 80 well sites, pump stations and other equipment with about 9,000 I/O points in its potable water treatment system with Open Secure Automation (OSA) units from Bedrock Automation. These provide PLC/RTU capabilities, and serve as proxy servers because of their intrinsic security and ability to work with an encrypted, Ethernet-based, peer-to-peer radio network. The new OSA controllers at 12 remote sites bypass the legacy concentrators, and report field data directly to the Bedrock proxy, but the concentrators will be removed once the remaining 74 sites are converted. Source: ECCV and Bedrock Automation
Because of traditional barrier concepts and locked-down mindsets, "open" isn't a word that usually goes with cybersecurity. This is where the physical world and its walls and locks fall short of covering the virtual world of software, networking and information technology (IT) that increasingly need openness to implement and maintain effective cybersecurity.
For instance, when the East Cherry Creek Valley Water and Sanitation District needed to upgrade its potable water treatment facilities and pump stations from their increasingly obsolete PLCs, its managers saw an opportunity to improve their cybersecurity, too. Located in Aurora, Colo., ECCV serves about 60,000 resident on the outskirts of Denver.
"Like most other public utilities, we must adapt to an ever-changing world and that includes cybersecurity," says Shay Geisler, I&C administrator at ECCV. "We’ve always had robust physical security, and required usernames and passwords for access to critical systems and controls, but we're seeing the world around us changing quickly. Many of today’s automation technologies aren't as secure as they could be because they were developed long before security was a major issue, so most of their security was added as an afterthought.”
Besides its old PLCs, ECCV used desktops and a dedicated Windows Server OS running its SCADA software with an OPC Server driver, which communicated with the PLCs via legacy Bristol standard asynchronous protocol (BSAP) and some Ethernet IP devices. Data concentrators above the PLC network manage data communications and aggregation over a serial radio network linking about 80 sites, running a mix of RTUs and PLCs handling about 9,000 I/O points. The radios worked like firewalls, separating the SCADA network from PLCs in the field, even though their signals weren't encrypted.
This article is one in the 2021 cybersecurity update multi-part series.
View the rest of the series here.
“We knew security couldn't be limited to only the SCADA software. There were too many downstream systems and assets that would present a huge vulnerability if left untouched," explains Geisler. "We determined the vast majority of these potential vulnerabilities could be solved by addressing the PLC and SCADA communications system.”
Upgrade to open cybersecurity
Geisler and his team focused on securing three communications paths: SCADA software to PLC, PLC to PLC, and the radio network. They investigated several strategies to secure communications, including adding firewalls and network cloaking, but they determined that achieving enough security required upgrading the PLCs, RTUs and radios. They worked with automation solutions supplier Process Control Dynamics and system consultant RSI Co., and selected Bedrock Automation’s Open Secure Automation (OSA) units to provide PLC/RTU capabilities and serve as proxy servers because of their intrinsic security and ability to work with an Ethernet-based radio for data encryption (Figure 1).
“We're slowly upgrading the remote sites one-by-one that were serviced by legacy data concentrators, and converting each to use the secure Bedrock controller," says Russ Ropken, VP and automation specialist at RSI, which developed the new architecture and its seamless transition at ECCV. "The new controllers at the remote sites bypass the legacy concentrators, and now report directly to the Bedrock proxy. Once all sites are converted, we'll remove the legacy concentrators.”
Because the OSA Remote units support BSAP, ECCV could continue communicating with its remaining legacy devices while transitioning to new controls, avoiding any significant service interruption. With the Bedrock units installed, ECCV's operators can employ new SCADA features that extended a root of trust from the PLCs to the HMI/SCADA system, which limits communications with the PLCs and other controllers to only securely certificated programs and users. This enables the utility to easily execute standard IT certificate practices, such as time limitation and revocation to individual users or groups. The result is secure, certificated communications from the SCADA software down to the remote PLCs and RTUs.
"If you solve a cybersecurity problem by building new from the inside out, as we've done with intrinsically secure Bedrock OSA, it will reduce other problems and costs. If you try to bubble wrap cybersecurity onto legacy devices, it will add costs," says Albert Rooyakkers, Bedrock's founder and CEO. "Now, C-suites are talking more about cybersecurity and risk management because ransomware and denial of service attacks are altering careers and restructuring budgets to prevent vulnerabilities and invest in appropriate technical responses. The good news is pragmatic and logical cybersecurity with open and secure technology is real, and Bedrock shows how by supporting the MQTT publish-subscribe protocol and the Open Process Automation Standard (O-PAS). We also have free integrated development environment (IDE) software, training and firmware support. We started with a mandate to drive engineering software costs to zero, and it will continue."
Software enables security
To gain its secure control capabilities, ECCV needed to upgrade its existing 32-bit SCADA software to 64 bits, which let it use the latest Windows Server and Windows-10 OS. “Just upgrading the software gave our IT and OT departments more confidence," says Geisler. "We also gained operational features and functions, along with new security features in the SCADA software. This addressed some of the security issues we had with our legacy systems, but it wasn't enough. We still saw possible security holes downstream from the SCADA system, and we wanted to address them."
Geisler and his team decided the most secure, cost-effective approach would be to connect the SCADA and control networks with a secure communications channel. Instead of a costly and disruptive rip-and-replace, they settled on a phased-in approach with Bedrock's OSA Remote control units in the role of proxy server between the data concentrators and the SCADA server. “A cyber-secure data concentrator functions as a proxy server, which secures communications from the SCADA software and the PLC network. Downstream, the data concentrator speaks BSAP or Modbus directly to the existing field unit, as well as some Ethernet/IP for smart devices,” explains Geisler.
The upgrade's next phase was to secure a direct connection between the SCADA software and the well sites, pump stations and water treatment facilities with PLCs and controllers with intrinsic cybersecurity, along with new Ethernet radios. Covering so many I/O points required using the OSA platform, which scales infinitely by adding five, 10 and 20 I/O control module racks, depending on the I/O at each site. In addition, the 64-bit SCADA software connects directly and securely to a peer-to-peer network of scalable OSA controls connected to the encrypted radio network. The intention was to complete the final architecture in five years, but COVID-19-related delays may extend this schedule. As it continues to build the system out, ECCV can keep the OSA Remote concentrator/proxy nodes in place or remove them because the system will remain secure to the field-level PLC/RTU devices due to its secure Bedrock controllers.
The result of implementing Bedrock's OSA platform was secure, certificated communications from the SCADA software down to the utility's remote PLCs and RTUs. The OSA Remote proxy units are switching over to a peer-to-peer network of infinitely scalable, secure Bedrock control units connected by an encrypted radio network. ECCV is presently running field data through 12 of its target sites, with about 74 left to be added.
“We expect the technology for industrial systems to be ever-evolving and improving. With this open architecture and technology, we can continually improve and upgrade, so we don’t have to face this type of wholesale transition again,” says Geisler, who adds that ECCV expects to get more than 30 years of useful life from its new PLC/RTU systems.
