“If all the producers used pub/sub, we could just change the IP address, point it to a new broker in the cloud, and have it up before lunchtime instead of in months.” Chevron’s Todd Anslinger on the use of MQTT to streamline radio communications.
In the Permian Basin, Chevron is commissioning facilities to separate and store oil and water, about one per year, each with three separators and six tanks. It’s also doing property swaps with other companies in the basin to take advantage of common infrastructure. As a result, it’s integrating about one system per week, each with similar equipment but often, different control systems.
“There are at least five things to integrate—PLC, local touchscreen, control room graphics, PI historian, and decision support center graphics—that must be done in series,” said Todd Anslinger, Industrial IoT Center of Excellence lead, Chevron. “Then, we have to verify every step to rule out human error. At the same time, human resources (HR) says we must double the number of facilities with just a 30% increase in head count.”
Anslinger spoke at Rockwell Automation Process Solutions User Group (PSUG) 2019, this week in Chicago.
Today, Chevron does separate integrations and security among those five things and maybe more, such as a flat file for production and a production database. “A different person is responsible for each thing, and each connection is a separate 1:1 relationship,” Anslinger said. “This takes more than 30 days, often six weeks per facility, not one week.”
Further, using a radio network with conventional poll-response, “We have to create and tear down connections for each communication with each device. With 16 remote sites, both analog and digital communications, it takes four minutes for a round robin. But with 50-100 devices per field office, it takes an hour, due to latency in the radio communications,” Anslinger said.
Publish/subscribe to the rescue
Instead, Anslinger proposes to use the MQTT (Message Queueing Telemetry Transport) protocol to lower bandwidth and speed integration. “Today, using a SCADA host doesn’t look too daunting, but in 50 business units we have 2,500 applications, each with 1:1 relationships,” he said. “How to manage all these starts to look like a nightmare.”
So, Anslinger proposes adding a MQTT broker to the SCADA host—a single point of contact. “Now, we can poll disparate devices—Modbus, etc.—as well as Rockwell Automation devices with native MQTT. For example, a pump pushes data to the host. The host is looking for ‘dot pump.’ Instead of step-by-step serial, the pump publishes and upstream systems get it.”
In the field, facilities might be 120 miles away, and across mountains. “It might take more than four hours to drive there—a big pain to go out there and physically manage the devices,” Anslinger said. “But with radio, it’s a big infrastructure cost to gain speed. The last mile is where we’re severely limited.” There’s a lack of bandwidth, and slow radios eat it up. If more frequent communication is needed for a critical measurement, say, to monitor a tank level, “You might have to shut down the SCADA to get bandwidth for that poll,” he said. “All the other information is lost.”
But how many variables change minute-to-minute? Some, but not all of them. Publish/subscribe (pub/sub) limits communications to just the parameters that change. “With Rockwell Automation PlantPAx objects, the initial poll might be 150 parameters, but then you only get the ones that change,” Anslinger said. “This frees up bandwidth, reducing it to about 15% of what it was before.”
It also reduces the time to integrate standard add-on instructions (AOIs). “If you make a change, everything is updated automatically,” Anslinger said. “A PLC change automatically filters up to the control room, all the way to PI historian. It just shows up.
“If all the producers used pub/sub, on a new or acquired facility, we could just change the IP address, point it to a new broker in the cloud, and have it up before lunchtime instead of in months, while we risk spills due to lack of information.”
Studies show speedy results
Anslinger’s Center of Excellence researched three scenarios comparing the time to propagate an AOI addition through the system: Competitor PLC/HMI (competitor PLC with a competitor touchscreen with a competitor HMI with OSI PI historian); Full vertical Rockwell (Rockwell CLX with ViewME with ViewSE with HistorianSE); and Sensia ConnectedProduction (CLX compute module, ViewSE running on VersaView, ViewSE in the control room, and a cloud instance of ConnectedProduction with VantagePoint).
“Before I show you the results, if you didn’t know anything about how poll/response worked, how would you want it to work?” Anslinger asked. “For example, consider an ordinary printer. You used to have to download a driver but now you just power it up, point it to your Wi-Fi, and it just works, you don’t have to integrate it. If you make a change, it just shows up. Pub/sub—MQTT—can do this.”
For each scenario, time measurements were taken to “add a tank object” to an existing system, and download to the controllers/computers:
- Competitor PLC/HMI: 22 minutes
- Full vertical Rockwell: 13 minutes
- Sensia Connected/Production plus mobile: 4.5 minutes
“In practice, since you still have to check for human errors, commissioning takes one week instead of two weeks,” Anslinger said.
MQTT in practice
Instead of pure MQTT, Chevron focuses on SparkPlug B (SpB), which adds features including birth certificate and death certificate (session awareness). “When you add a new PLC or AOI, you get a birth certificate. You see that, and you can pull it in,” Anslinger said.
Without Session Awareness, you might not know if an instrument died. For example, at one point, “Lightning took out a level gauge, and we didn’t find out until a local driving by called Chevron about water running across the road,” Anslinger said.
“With standard MQTT, you only get so much, and you have to add to it to build a structure of tag names to know what you have,” Anslinger said. SpB accounts for the hierarchy of Chevron facilities, including a tagname convention that helps with contextualization of data. This has been given to the Eclipse Foundation, which is somewhat like ODVA and is managing this “open standard” with other oil & gas companies, he said. “Other vendors have already implemented SpB, and Rockwell/Sensia has programmed a compute module that is SpB.”
In summary, pub/sub is good for adding similar facilities when you use standard objects and dashboards. “You can just add your standard object,” Anslinger said. “Additions and changes can automatically show up through all levels of the process control network, and even business systems.”
Pub/sub pushes updates to the broker only when a value changes outside a set threshold. This uses about 15% of the bandwidth of poll/response, thus opening bandwidth for other technologies from condition monitoring to video.
Finally, Anslinger said, “Make sure you have session awareness so you are not running blind.”
