Critical infrastructures cannot be secured because network security and engineering won’t work together

There continues to be a gap between the engineering organizations responsible for reliability, functionality and safety on the one hand, and the network security organizations responsible for network security on the other. This was evident in the recent White House Energy Modernization Cybersecurity Implementation Plan and the associated comments on a LinkedIn post on the use of webservers in instrumentation systems.

Control systems are neither just engineering nor network security but a combination of both: modern networking technologies provide enhanced productivity and efficiency but often at the expense of cybersecurity vulnerabilities. Yet both organizations continue to act as if the other doesn’t exist. The term operational technology (OT) makes this problem worse, as its focus is on networks and not on the issues uniquely associated with control system field devices.

Get your subscription to Control's tri-weekly newsletter.

The problem starts at the university or junior college level where cybersecurity is taught within computer science, yet most schools do not require the computer science students to take an introductory course in engineering. The converse is the engineering disciplines of electrical, mechanical, chemical, nuclear and systems engineering do not require students to take an introductory course in cybersecurity. And from there, the die is cast.

The following are examples of cases in which engineering organizations designed equipment with cyber vulnerabilities, and network security organizations created reliability and safety issues with control systems and facility equipment.

Engineering without network security input

In the early 2000’s, heat waves caused failures of many electric distribution transformers. Consequently, one of the substation suppliers published the following advertisement dated Aug. 29, 2005: 

“Equipped with an Ethernet interface and Webserver, Vendor A Unit Substations now provide simple, affordable access to power system information – including transformer coil temperatures – using a standard Web browser. The pre-engineered equipment connects to a customer’s existing Ethernet LAN much like adding a PC or printer. Unit substations include a Temperature Controller, which provides remote access to transformer data, in addition to its primary role in controlling cooling fans. With a simple click of a mouse, it is easy to monitor transformer coil temperatures per phase and verify colling fan status at a glance. These new capabilities make it possible to correlate circuit loading with transformer temperatures to extend equipment life. The Ethernet interface and webserver are embedded in this vendor’s medium and low voltage switchgear, unit substations, motor control centers. switchboards, and panels.” 

Are these scenarios “secure by design”?

1. Currently, Schneider Electric provides standard equipment, the controller with an embedded Webserver with a predefined, built-in website. You can use the website for module setup and control as well as for application diagnostics and monitoring. These pages are ready to use with a web browser. No configuration or programming are required. The web server can be disabled. The webserver is a tool for reading and writing data and controlling the state of the controller, with full access to all data in your application.

However, if there are security concerns over these functions, you must at a minimum assign a secure password to the webserver or disable the webserver to prevent unauthorized access to the application. By enabling the webserver, you enable these functions. The webserver allows you to monitor a controller and its application remotely, to perform various maintenance activities including modifications to data and configuration parameters and change the state of the controller. Care must be taken to ensure that the immediate physical environment of the machine and process is in a state that will not present safety risks to people or property before exercising control remotely. 

2. In 2021, more than 3,000 new smart instruments that had no passwords, even by default, were installed in a petrochemical facility. You can’t rip and replace these sensors, as these new sensors themselves were the “replacements.” 

3. I reviewed the 2023 instrument data sheets on digital pressure transmitters from four major US and international process sensor manufacturers. All four of the vendors are actively involved in industry cybersecurity activities. The data sheets were more than 70 pages.

Consequently, I did a word search on the following terms: cyber, security, passwords, authentication and encryption. The four vendors’ data sheets did not mention any of those terms. On the other hand, I did a word search on the word “remote.” That term was used extensively as all four vendors support remote connectivity. Or as a colleague stated, engineers will pay extra for remote access without considering the cybersecurity issues associated with that capability.

Additionally, in one data sheet, Bluetooth was enabled by default assuming that distance will mitigate any cyber vulnerabilities – a questionable assumption at best. In fact, in the June 2023 issue of Control Global – “Updated pressure transmitter increases technician safety and makes work faster and easier" – powerful features like graphical back-lit display, Bluetooth connectivity, easier to navigate user interfaces, level and flow specific configurations and diagnostic features allow you to perform commissioning, maintenance and troubleshooting tasks faster than ever”.

4. A Control.com technical article dated April 22, 2024, was on “Setting IP address and networking parameters on a PowerFlex VFD.” According to the article, “the PowerFlex 52x series has a removable faceplate and can be connected to a computer with a USB cable. This should be done before powering up the drive with AC voltage. Plug the USB cable into the drive, and a small application will open, allowing you to upload a drive config file to your computer. Open this file with Notepad (or a text editor), adjust the correct parameters, and then use the included application to re-download the parameter list. This process is simple, but it can save a lot of time when commissioning many drives all at once.” Notepad and other text editors have no cybersecurity capabilities.

5. Dec. 12, Emerson’s Jonas Berge issued the following blog – “Are webservers the future of instrumentation?” The blog demonstrates the divergent needs of engineers for reliability and efficiency and OT cybersecurity for cybersecure operations. As Jonas states:

• Accessing a device from a web browser is marvelously convenient. Managing a hundred devices by periodically browsing them one-by-one is a nightmare.
• To manage large number of devices – a fleet of devices – you need management software.
• The recommendation is to deploy devices with embedded webserver but also a device management system.
• The result is the convenience of a system monitoring the fleet of devices combined with the convenience of browser access to an individual device.

Yet, as Emerson’s Michael Lester stated: “L0/L1 Devices and instruments may not have the same capabilities as higher-level devices from L2 North and will still have dependencies on defense-in-depth and rely on the security capabilities that are built-in and on the adjacent devices and systems similar to today’s architectures.”

Emerson and Schneider Electric are engineering companies whose primary job is to maintain reliability and safety whether from malicious or unintentional events. Erroneous unintentional actions such as inputting a wrong address or setpoint can cause the same or similar impacts as malicious cyberattacks.

Finally, an acknowledged process industry instrumentation expert stated: "I have spent years talking to brick walls and brick heads about the lack of security in field devices. Their response is typically that they are air-gapped and that everything is safe and secure. Irrational fantasy at best. I am not alone in this quest, but I am definitely in a minority.”

Network security without engineering input

A large utility performed security scans of several critical substations. The security group was scanning data center assets. Because the effort was successful, the security group decided, with no input from operations, to extend the scanning to NERC CIP substations, starting primarily at the 230/500KV level. The security group had no previous experience with scanning substations. No notification was given about the scanning change to the internal support groups that were responsible for this function. The OT team was notified that substation scanning was started with a new security port scanning tool.

Following the scans, the relays showed trouble, but the DNP (OT communication protocol) polling was working properly and the networks in most substations were stable – SCADA was unaware of the problems. The port scanning by this tool caused the real time protocol operation of the relays (IEEE61850/GOOSE) to stop and suspend operation at the CPU (two different relay suppliers) but left the DNP/non-real time operations alone – the worst possible circumstance.

To clear the trouble and restore operations, each relay had to be cut out and rebooted. More than four hundred high voltage relays were affected. All the devices in each substation were affected at the same time in every case. Without knowing that a security scan was initiated, it looked like a DDOS attack resulting in equipment malfunctions. Given the appropriate circumstances, this IT-caused problem could have created a major western states outage. 

Because of the number of facility shutdowns from inappropriate patching practices on control systems, the International Society of Automation (ISA) initiated a recommended practice/standards effort to develop patch management for control systems. 

On Mar. 16, 2022, NIST issued NIST Special Publication 1800-10 “Protecting Information and System Integrity in Industrial Control System Environments: Cybersecurity for the Manufacturing Sector.” The NIST report states, “In this project, the focus was on the engineering workstations and not on the manufacturing components. It is acknowledged that many of the device cybersecurity capabilities may not be available in modern sensors and actuators.”

The White House Energy Modernization Cybersecurity Implementation Plan issued December 2024 included “Section 8 Distributed Control Systems (DCSs)”. The term “DCS” is an accepted term that has been used in the process and power industries since the 1990s. However, according to Emma Stuart, “the term DCS has been applied in lots of publications now to things that involve DER, VPP, BESS, ADMS and other plant and mass orchestration systems. It makes far more sense than just using DERMS as it describes a wider swathe of technology. Lots of mass orchestration is using it and that’s the point. There’s lots of words and phrases in this space that have dual meanings now also.” The security organization creating new terms will not sit well with the engineering community.

The lack of cybersecurity capabilities in legacy control systems means that CISA’s cybersecurity recommendations such as system hardening, removing default passwords, patching and penetration testing that are relevant to internet protocol (IP) devices can be harmful to process sensors and actuators that don’t have the IP stacks and other security capabilities to utilize the CISA recommendations.

Antivirus software has shutdown older legacy control systems. Addressing Microsoft tools and scripts are irrelevant to process sensors and actuators that don’t use Microsoft operating systems; of which there are many. There have been numerous cases where inappropriate IP network technologies and testing have shut down or damaged control system devices which is my concern about CISA’s guidelines.

There is a plethora of “OT experts” who know little about control system devices yet provide OT/ICS advice. Control system field devices are technically and operationally different from IP networks. Consequently, this inappropriate advice from these “OT experts” can and has harmed control system field devices and shut down facilities.

Conclusions

The engineering and network security personnel need to understand each other’s strengths and limitations. This needs to start in college. Control system cybersecurity training is needed to minimize the inappropriate advice being dispensed by “OT cybersecurity experts” who don’t understand control system field devices. Senior executives need to ensure that the engineering and security organizations support, not ignore each other – that is not happening.