If it had a face—and it sort of does—would you recognize supervisory control and data acquisition (SCADA) if you saw it on the street? Would SCADA even recognize itself in a mirror these days?
Chance are, probably not. This is because all the earth-shaking shifts due to data digitalization and the Internet are also turning SCADA on its head, and taking it in entirely new directions. Granted, its conceptual borders were always pretty porous—some water/wastewater users think of its their entire control system, while others consider it limited to just their human-machine interface (HMI), However, even as it continues to display collected information in many applications, the accelerating influence of cloud- and virtualized-computing services and the Industrial Internet of Things (IIoT) are pulling SCADA into strange shapes to perform unfamiliar tasks beyond its traditional jurisdiction.
"Where utilities used to have second and third redundant control and SCADA systems for comprehensive disaster recovery, they're now synching with duplicate systems in the cloud," says Michael Chmielewski, offer management VP, process safety and SCADA, Schneider Electric. "We have one large, U.K.-based gas utility that's talking about moving its whole SCADA system to the cloud. SCADA isn't going to dissolve, but it's going to become more of a secure service deployed out of the cloud."
Seeking definitions
"Even as a broad term, SCADA is not a distinct thing anymore. Now it's all about the data, and how what used to be SCADA is converging with and incorporating a whole bunch of new technologies," says Chuck Tommey, P.E., business development manager at A&E Engineering in Greer, S.C., a member of the Control System Integrators Association (CSIA). "On-premise SCADA with servers in racks, Ethernet to PLCs and HMIs are still perfectly valid and will continue for a long time, but we're on the brink of a big mindset change.
"We've all heard about the cloud, IoT and data analytics for 10 years, especially from the information technology (IT) side, but now they're converging with operations technology (OT) on the plant floor. Virtualization, IIoT, cloud computing and data analytics are all part of SCADA, or they can be used to enhance or manage it. This means the most successful end users and companies will be those with well-integrated IT/OT teams or IT departments with significant understanding of OT."
Bob Ell, senior marketing director, oil and gas, Honeywell Process Solutions, adds that, "SCADA technology has been changing for many years, adding PCs, new sensors and data centers, but even though the amount of data coming in now is just crazy, the challenge is still getting something meaningful out of it, and turning data and information to wisdom.
"The SCADA journey is different for each customer depending on their different requirement and levels of sophistication. IT may view it one way, but traditional automation and SCADA hasn't changed the way it connects to distributed assets and controls. Water/wastewater, oil and gas, and power applications have done SCADA for a long time, but now they can use an understanding of the Internet to remove more costs. Likewise, they can use cloud technology to more easily display better information to operators, and help them do a better job, collaborate more, and run their businesses more effectively."
New sandboxes, new shovels
Logically, because the cloud, virtualization and IIoT multiply the connections a SCADA system can have with other systems, they also increase how many different functions it can take on—along with the tools to perform them.
To monitor and control more than 200-megawatts (MW) of utility-level, solar generating capacity at five plants built by Depcom Power in Scottsdale, Ariz., Vertech, a CSIA-member system integrator in Phoenix, Ariz., developed an innovative SCADA system that could handle each facility's thousands of connected devices from many vendors. They include 15,000 tags, 30 screens, 10 clients, 10,000 alarms, one local Microsoft SQL database, one offsite database and 2,000 historized tags. Besides coping with the volume of tags, Josh McGuigan, Vertech's senior control systems integrator, reports that Depcom's new SCADA system also needed to be robust to cope with new plant rollouts and data levels that can quickly become overwhelming.
"For the solar power plant SCADA system at Depcom's plants, we used Inductive Automation's standard Ignition software architecture, including one local historian and one connection to a database in the cloud," explains McGuigan. "On a typical site, an Ignition gateway will be directly connected to nearly 100 devices. However, some of those devices act as gateways themselves, so the total system is used to monitor and control around 3,000 devices, which amounts to more than 15,000 I/O tags per site."
McGuigan adds that intelligent reporting, which does more than provide the top 1,000 rows of a database table in a tabular format, is essential for large solar arrays. Using advanced scripting the Ignition Reporting module, Vertech designed a report that analyzes data from hundreds or thousands of strings of solar panels to highlight low-performing equipment and prioritize operations and maintenance team activities.
"Ignition lets us provide a SCADA system for Depcom that improves on the industry's status quo," adds McGuigan. "Users now have more data available and their interface is easier to use, allowing operations and maintenance teams to be more effective in identifying and troubleshooting issues. The analysis of site performance data, which used to be a manual task, is now automated, so any site performance issues are quickly brought to the appropriate people. The system can also be developed and commissioned in record time, allowing us to meet schedules that would otherwise be impossible."
Titus Crabb, P.E., president of Vertech, reports, "We work with Siemens, and they call SCADA the 'digital glue' that connects everything. We agree that SCADA connects the application layer with Internet-based services, which lets it sit between data reporting, dashboards, maintenance, overall equipment efficiency (OEE) and enterprise resource planning (ERP). However, this also lets OT and IT understand each other better, use common databases and tools, know how data flows between the plant and enterprise, break down fear, and learn to trust their network."
Travis Cox, co-director of sales engineering at Inductive, adds that, "SCADA is integrating with third-party applications like web services, business intelligence, machine learning, SQL databases and others. However, they're not just looking at intelligence, but are becoming more homogenous and bigger, and their business is getting more involved, too. In fact, we're now on the cusp of taking previous polling protocols to PLCs, and using new message-oriented middleware and MQTT publish-subscribe protocol to connect devices, and let any application subscribe to them without needing to go through the usual SCADA system. This new MQTT and IIoT architecture gets SCADA back to the mission-critical role it had in the first place."
Dealing with (big) data
So what can cloud-enhanced SCADA do with its new powers? Probably the most important job is make sense of the buckets of data coming in from all its new connections.
For instance, the reticulated water supply system in Hamilton, New Zealand, consists of one treatment plant that sources water from the WaikatoRiver, and delivers it through more than 1,000 km of pipes to eight reservoirs and more than 150,000 residents in more than 51,000 homes and businesses (Figure 1). The city also runs the Pukete Wastewater Treatment Plant (WWTP). To comply with drinking water standards that New Zealand revised in 2008, including retaining operating records for 10 years, the city had implemented Rockwell Automation's RSView 32 SCADA system 10 years ago, but its manual data recording to Microsoft Exel spreadsheets recently needed upgrading.
[sidebar id =1]
“Our previous system was outdated and we required an upgrade to help simplify the process of complying with current water regulations in New Zealand," says Gary Pitcaithly, automation and electrical manager at Hamilton City Council. "Not only that, but we identified the potential for improving operational efficiencies at the plant by implementing an integrated system that aims to increase productivity and reduce downtime.”
As a result, Hamilton implemented Rockwell Automation's latest FactoryTalk software, including its FT View SE, FT Historian, FT Vantage Point, FT Asset Centre and FT ViewPoint, which provides real-time exchange of information throughout applications and organizations for improved business decisions, responsiveness and productivity, reduced costs, and easier regulatory compliance with long-term data storage and automatically generated reports. Staff can also externally manage the system via tablet PCs or smart phones.
“The upgrade has delivered greater ease of use of our system throughout the WWTP," adds Pitcaithly. "The new historian is superior in how it stores data, and makes generating information for vital reports much more efficient. Vantage Point lets us develop reports at will, whether for compliance to water standards or for other needs. They can then be published as web-based reports that are available for anyone authorized to view them.
"FactoryTalk has enabled our team to be more flexible with their time, as we're now able to edit or update reports as we go. We simply store our data directly into Historian and the data spreads directly from the programmable automation controller (PAC) to a human interface. This data is incorporated into spreadsheets for us to interrogate, whether it's on a daily, weekly or monthly basis, to tell us if we've had a breach in turbidity or if chlorine levels aren't what they should be.”
Alan Cone, WinCC marketing manager at Siemens Industry Inc., adds that, "The amount of data coming off devices on the plant floor is the biggest trend affecting SCADA today. The data acquisition piece of SCADA is becoming more important, and plant owners continue to reduce downtime and improve overall efficiency of the plants to compete in a global market. As equipment supplies add more intelligence on the field level devices, this creates lots of data on the health of the operation. This data can be overwhelming if not properly managed. While control is still a key part of the SCADA system, more importance is being placed on the data aspect. In the past, the data was mainly used for reporting needs. Now, the data is used to refine and adjust the running plant in real time.
"Understanding and interpolating the data from the field/factory is very important. WinCC has a historian built into the core system, so an additional package isn't needed to log the data. Siemens also has a reporting tool called Information Server that allows users to access the data in WinCC through a web portal, and write a user-defined report that can bring up-to-the-minute data to the requester. Our Process Historian archives data in WinCC on a separate historian for offsite, long-term storage, or as a single repository to collect data from several WinCC SCADA systems at various locations. Information Server can also be used to report on the data in the Process Historian."
Affordability aids adoption
Another of the unglamorous—but still crucial—advantages that cloud and virtualized computing add to SCADA systems is the fact that they can quickly help reduce operating costs after a relatively small investment at the beginning.
For example, to help smaller and rural water/wastewater utilities perform data acquisition, alarming, reporting and other SCADA functions on a budget, system integrator Perceptive Controls in Plainwell, Mich., recently developed its Perceptive Polaris cellular, cloud-based SCADA system and software, which employs SNAP PAC controls from Opto 22, and avoids using costly servers and hardware.
One of the key development challenges Perceptive's engineers faced was how to reduce data sent between lift stations on the SCADA network. “We knew that using cellular modems meant one of the most important requirements of this project would be the ability to
transmit the smallest data packets possible, with as much data in each packet as possible,” says Kevin Finkler, software engineer at Perceptive Controls. “We had to stay under the data caps of the cellular provider we planned to use.”
The system integrator first tried posting data from a controller to a cloud-based server, but testing showed this method was too slow, and couldn't send configuration changes back to controller. While considering alternate options for transferring data, Perceptive's developers investigated SNAP PAC's Representational state transfer (RESTful) application programming interface (API) capabilities, which include a built-in, secure HTTP/S server with an open, documented API that creates a RESTful architecture. RESTful and its technologies, like HTTP/S and JavaScript Object Notation (JSON) are intrinsic to IoT and essential for web, data and mobile-based application development.
“After switching to the new RESTful API method, we now have a cloud-based software application running on a dedicated server that uses SNAP PAC’s RESTful API to request data directly from the controller, explains Finkler. “Requests are made over a private cellular network to avoid cybersecurity concerns, and avoid opening ports in firewalls. We store data in float tables on the PAC (about 44 indexes per table), and the software can grab up to 100 tables of data per request without slowing down communication performance.”
Perceptive's cloud application then uses the RESTful API to write back how many tables were retrieved, so the controller can delete the old data, and move everything up in the table with new data again at the top. This ensures that all data is received into the cloud application. “It’s more efficient to make the cloud application process large amounts of data, instead of making the controller do the work in addition to its normal operations,” adds Finkler. “This method saved an average of 5.8 kb per data set transmitted, which ended up saving us about 250 MB per day, adding up to significant savings in cellular data charges.”
A&E's Tommey adds, "We see a lot of lower-cost SCADA solutions at shows, but when you get down to it, their initial cost is still high for users. However, when OT and IT converge in the cloud, they can reduce SCADA and other costs, and turn many capital expenses into operating costs. Plus, today's subscriber-based fee structures can reduce costs even more, especially at the front-end where these expenses can be hard for users to swallow."
For instance, Tommey reports that a typical SCADA project with three servers, wired and wireless networking, field devices and software licensing can add up fast to an average of $100,000, while an equivalent, subscriber-based version on the cloud may only cost $10,000 upfront with additional subscription fees on a monthly or annual basis. "This is a lot more approachable for users in their local budgets," says Tommey.
Despite this tenfold cost reduction, Tommey adds that many potential users remain reluctant to do SCADA in the cloud due to security concerns, worries that Internet links will go down, and anxiety that their data won't always be available to them. "As connectivity gets more reliable, more users will make the switch," he says.
To help users handle the transition from traditional SCADA to the cloud, Tommey adds that A&E is now designing hybrid systems with local, secure, second-by-second data storage using traditional HMI and historian software (such as Ignition, Wonderware, WinCC, OSI Pi or FactoryTalk), and then sends only certain pieces of data to the cloud for analysis. The cloud now allows this data and the analytical results to be more readily available via tablet PCs and smart phones. "This enables some remote data access, distributed alerting, machine learning and prescriptive maintenance without putting everything in the cloud," Tommey adds.
Tom Buckley, IoT business development manager at Iconics, adds that, "We think the biggest value proposition of bringing SCADA together with virtualization and the cloud is that they can bring together OT and IT people and technologies, and really bridge the gap between those formerly separate silos. In addition, once data is published to a cloud service and analyzed, smart gateway devices can bring intelligence and better algorithms back from the cloud to the edge to help operations strengthen predictive maintenance, making it proactive instead of reactive, reducing downtime, and improving overall efficiency."
Buckley adds that intelligent devices can distribute instructions to the edge with assistance from solutions, such as Iconics' new IoTWorX software, which supports multiple operating systems such as Microsoft Windows 10 IoT Enterprise and Windows 10 IoT, as well as Linux embedded operating systems like Ubuntu and Raspbian. "Our software can go into third-party manufacturing IoT gateways, attach to that hardware portfolio, and take high-end, cloud-based software and bring it to the edge for secure analytics and visualization," he explains.
Ongoing interface evolution
Surprisingly, once SCADA and the cloud—and their users—get familiar with each other, they begin to generate subsequent new capabilities that weren't apparent at first.
"There's been a move away from panel-mount SCADA because there's not as much reason to have local HMIs, but even after the initial jump to tablet PCs and smart phones, they were still tied to proprietary software," says Will Aja, customer operations VP at Panacea Technologies Inc., a CSIA-member system integrator in Montgomeryville, Pa. "More recently, we're seeing greater use of thin clients using software like ACP ThinManager that performs processing in the cloud, and serves up screens to workstations and their users wherever needed, which is incredibly convenient and provides a lot of functions." ThinManager is a division of Rockwell Automation.
"Panacea provides a service to migrate users from hardware SCADA and HMIs to virtualized infrastructures, and we offer it on all new projects, which can include hybrid, onsite clouds and thin clients," explains Aja. "Having a virtual infrastructure like this means you're set an don't have to worry about hardware updates for years because it's so easy and cheap to replace the monitors. It's also more secure because users can set up who gets access via a Windows domain, and they can also do geo-fencing or location resolving, which only grants access based on a user's physical location. This makes is possible to design a very secure system that only allows access to its components when a user is next to them, but terminates their session if they walk away, which means remote access isn't allowed."
Beyond saving with thin clients, Aja reports that virtualization also make it easier for SCADA systems to perform less-costly testing and performance checks. "Many applications and plants do tests and testbeds to check performance," he explains. "It's like putting together an orchestra for a concert: you have to make sure each group is doing what it needs to. However, this can be a lot easier with virtual tools and a virtual development environment. These let users do their code ahead of time, and have what they need to migrate their application. We recently did a pharmaceutical process plant project with 70 HMIs, 34 servers and 45,000 tags in a facility running 34,000 recipes per year, and these kinds of virtual preparations let us migrate its whole SCADA structure in about two hours."
Likewise, Aja adds that enhancing SCADA with virtualization can impact every part of a business. "IT has more control over who can access the system; operations can simplify training; accounting can get rid of inventory by standardizing on devices; and plant management that previously had to migrate SCADA equipment every three to five years can extend their lifecycles to 10-15 years," he says. "In the future, I think we're going to integrate even more with virtual tools for SCADA. We've already signed up to maybe be a Google Glass partner, which should let the operators walk around, see highlighted OEE data for equipment in front of them, and maybe fix it more easily. Plus, all the disparate SCADA technologies, such as alarm annunciation, data delivery and others, are going to going to come onto a common, standardized platform."
Michael McEnery, president of McEnery Automation, which is a CSIA-certified system integrator in St. Louis, adds that, "SCADA vendors are realizing that users have to get the costs of implementing projects down. This means system integrators are becoming a bigger part of the equation because, while software costs have remained about the same or decreased, hourly rates and labor costs continue to increase. As a result, SCADA suppliers are trying to provide functions that work right out of the box, such as preconfigured PID faceplates and graphic objects with coordinating libraries of PLC function blocks. These tools really reduce our implementation time. I think our time to deliver an average project is approximately 50% less over the last 10 years."
Security underpins everything
While the many benefits of cloud-enabled and IIoT-aided SCADA are terrific, all these added connections to higher-level networks and the Internet come with increased risk of probes, intrusions and attacks, which demand updated cybersecurity and constant vigilance by staff.
"The big bugaboo impacting all these new technologies is cybersecurity, especially in the OT space," explains A&E's Tommey. "Cybersecurity must be addressed as a continuous cycle, but a lot of companies on the OT side don't understand this yet. They all did 20-30 years of continuous work on lean process improvement, and that continuous improvement thought process needs to be extended to cybersecurity."
Tommey reports that users must start by knowing themselves, their networks and everything that's attached to them. Next, they must learn to understand the threats out in the larger world, such as the currently active viruses and their attack profiles, and use this knowledge to determine what they must do to harden their individual networks and applications. This can include updating passwords and security polices, implementing network segmentation and firewalls, following security standards like NIST 800 and ISA/IEC 62443, and constantly monitoring network traffic with security information and event management (SIEM) software as a first step and intrusion detection system (IDS) and intrusion prevention system (IPS) software as a second step.
"The problem now is that this security monitoring piece doesn't exist in 95% of manufacturing facilities because they just have traditional air-gapped equipment or installed simple firewalls. There's more security monitoring on the IT side, but there's very little on the plant-floor side, and many small start-ups and larger suppliers are seeking to fill these gaps."
