The Sewerage & Water Board of New Orleans runs an independent, historic, 25-HZ electrical generation system powering the city’s water and drainage pumps installed more than 100 years ago. The city has eight power grid connections to the rest of the state, which were knocked out for weeks by hurricane Ida two years ago. Without its independent power, New Orleans would have seriously flooded during Ida, and there wouldn’t have been water for hospitals and fire protection.
“We were helping with generation system repairs and an upgrade, but our assignment changed when the city was also hit with a ransomware attack that knocked out its purchasing and other critical business systems for a couple of weeks,” says Stan Prutz, PE, president and CEO of system integrator QDS in Baton Rouge, La., a 20-year-certified member of the Control System Integrators Association (CSIA). “Even though this was an IT-based business system, the city was concerned that the OT-based controls and equipment running its power plant could be vulnerable, too.
Picking up speed
Consequently, when the city began a program to renew its power-generation system, security became a high priority. QDS was enlisted by engineering consultant Jacobs Engineering and prime contractor Walter J. Barnes Electric to complete detailed design and implementation of more secure controls and networking for SWBNO’s power plant supporting water supply, drainage and sewer treatment processes. Their improved solution relies on software-defined networking (SDN), and was implemented over two years and commissioned in 2022.
“Electrical utilities typically need network speeds that can react in a few AC cycles, approximately 4 milliseconds (ms), and typically use DNP, IEEE 61850 and GOOSE protocols,” explains Prutz. “The architecture Jacobs planned needed higher-speed communications because it uses peer-to-peer networking between intelligent electronic devices (IED), mostly protective relays. Circuit breakers must be able to open quickly to handle faults. These jobs are traditionally done with hardware, but it wasn’t flexible enough in this case, so Jacob’s concept was to open the breakers with generic, object-oriented, substation event (GOOSE) messaging defined by the IEC 61850 standard via high-speed Ethernet between two IEDs.
The utility decided to install 751 high-speed protective relays from Schweitzer Engineering Labs to protect pump motors, feeders and transformers. For the IEDs, SWBNO also adopted SEL’s real-time automation controllers (RTAC), which are similar to PLCs, but can run at higher speeds, serve as data concentrators, and support multiple protocols including IEEE 61850, DNP, Modbus and Ethernet IP. Many electric utilities and pipelines are increasingly remotely controlled, which has driven adoption of write-by-exception protocols like DNP that can tie them together, and run faster over broadband than the telephone modems they previously used.
Secured by software
To secure the new RTACs and relays in SWBNO’s citywide Ethernet network, Prutz reports that QDS implemented an SDN. “SDN technology is the latest hot thing for utility cybersecurity because the only data that gets routed is what you preconfigure with the flow controller. The flow controller defines how all the software defined switches (SDS) and other devices will communicate with one another on the SDN,” says Prutz. “SDN solutions provide centralized management and orchestration capabilities, which can simplify network security operations. Centralized control allows for consistent security policy enforcement across the network, making it easier to implement and manage security measures. An SDS switch looks like a regular Ethernet switch, but its software defines everything that routes between ports. There are no communications that aren’t predefined and preauthorized. And, if any extraneous communications show up—such as hackers probing for an open port—they’re revealed right away.”
This capability stops communications and data from being intercepted and possibly probed. However, the price of greater control is that it requires extra administration. For example, the flow controller software that configures the SDN must be paired with an intrusion detection system (IDS) to automatically send alerts about anomalies. The flow controller must be configured to learn what normal operations and acceptable traffic looks like. This isn’t particularly difficult, but it takes time, just as adjusting MAC addresses for hardware changes must be documented and accounted for in the flow controller software.
To get their cybersecurity infrastructures in place from the beginning, users must start with passwords, authentications and accountability for each user at each station, proceed to network segmentation and monitoring, and even address coding requirements for supporting multiple devices. These and others directives are part of the ISA/IEC 62443-3-3 standard (Figure 1).
VPNs and data centers add value
Paralleling its New Orleans project, Prutz reports that QDS also recently helped Slidell, La., upgrade the networking and monitoring for 115 sites in its water/wastewater system and reduce its administrative tasks, but this modernization also made it more secure.
“Slidell adopted cellular networking early on, about 15 years ago. So, when it came time to upgrade recently, they liked the idea of putting it all on the web,” says Prutz. “This meant that all the data from their Verizon cellular network with hardware VPN routers and encryption could all be on one large, secure, private network.”
Consequently, the municipality brought all its water/wastewater information to a virtual server-based, commercial, Tier 1 data center, operated by DartPoints and staffed 24/7 in Baton Rouge. This setup includes integrated Fortinet cybersecurity software between the data center and the Internet feeds to it. “This put the SCADA system in the data center, too,” says Prutz. “This lets the client use web-based Ignition SCADA software running as a virtual machine on data-center-redundant hardware to view all their data and receive alarms on their smart phones, and saves on hardware and maintenance because everything in the data center is included as part of a monthly site cellular fee.”
Using its new VPN routers and Internet-based data center, the city is projected to save 25% in upfront costs and 25% in ongoing, lifecycle costs over 20 years, compared to the expense of a traditional cellular-based SCADA system tied to an office-based operations or maintenance center onsite.
“A regular SCADA system with PCs in local facilities would require setting up and paying for individual cellular accounts for each site, which would’ve been more costly to maintain, and wouldn’t be as cyber-secure,” explains Prutz. “Our more cost-effective option links cellular modems with integrated VPN routers to a hardware VPN router located at the data center. This lets the city continue to use its standard Allen-Bradley Micro 800 PLCs as the brains of its pumping stations and other sites, remotely accessing and monitoring everything from the data center via the VPN, and allowing maintenance personnel to directly receive alerts and respond when a station has a critical alarm. Using the VPN not only means less cost and traffic than a traditional cellular network, but it’s also more secure because QDS can use Fortinet at the data center to manage cybersecurity for our client, which includes getting weekly anomaly reports.”
Beyond boosting security, Prutz adds that Slidell’s VPN-based, virtual data center also improves the utility’s uptime because it can get alerts to users’ smartphones quicker, and schedule repairs or maintenance sooner.
“This utility didn’t track SCADA uptime as much before, but lately it hasn’t experienced an outage in the year since this data center implementation was installed,” adds Prutz. “Users can graphically see each site’s operations on their smartphones at any moment, instead of traveling to their operations center to view a central SCADA system monitor. Now, the virtual machines implemented at the data center and QDS are responsible for central SCADA operations on a subscriber basis, including making sure that all required hardware, software and networking stays functional and up to date.”
Tailoring protections
Prutz reports that monitoring each water/wastewater site costs Slidell about $40 each per month, saving 25% overall due to reduced communications traffic and costs, and less maintenance, reconfiguration and replacement costs for SCADA hardware and software.
“The right solution usually depends on the size of the organization,” says Prutz. “Large industrial and municipal users have significant IT departments with established cybersecurity standards. We work within their existing frameworks, and adhere to their methods. If shutting off the wrong pump could shut part of the city down and cost millions of dollars, higher security is required. Smaller industrial and municipal users have less structured IT departments and are less likely to face these risks. These organizations require us to consider their circumstances and provide cybersecurity to the best practice level.
“Big oil and gas or petrochemical companies typically have tight cybersecurity policies based on U.S. Dept. of Homeland Security (DHS) requirements. Public utilities have to follow the same rules, especially because the big picture is always evolving and the individual cyber-threats are always changing. For example, ransomware wasn’t a huge problem 10 years ago, so it’s important to monitor these changes from a current perspective, and be ready to respond.”
