“The biggest impact from my end is that all the blocks in the control processor will be initialized.” Kessler Industrial Controls’ Maks Wilde shared his firsthand experience with online controller upgrades at Schneider Electric Innovation Days this week in Austin, Texas.
Process interruption is costly. Planned shutdowns need to be well-organized and swift. Any halt in production means lost revenue, so the ability to update equipment without interruption is very attractive.
EcoStruxure Foxboro DCS online upgrade (OLUG) functionality is available in two modes. Because the cold-start version reinitializes everything, it’s not particularly useful, explained Tom Rosborough, program manager, research and development, Schneider Electric. He presented with Maks Wilde, senior control systems engineer, Kessler Industrial Controls, on equipment upgrades without process interruption at Schneider Electric’s Innovation Days this week in Austin, Texas.
“When people say, ‘OLUG,’ they mean a warm start,” said Rosborough. “You initiate the OLUG, and the image is sent to the shadow control processor (CP). Now we can transfer the checkpoint and calculate outputs based on the checkpoint data. During this time, the primary CP is still controlling the process.”
Know your limits
When you’re doing an update online, you’re going to want to wait for stability, explained Wilde, who shared some of his firsthand experience with the process at Kessler.
“There are known limitations of an OLUG warm start,” he cautioned. “There is a delay from the time that the primary CP creates a checkpoint and copies it to the shadow CP and when the shadow CP installs it. That can be from 0.5 to 15 or more seconds and can lead to a stale checkpoint. For an FCP270, it can be up to 60 seconds, so it could be almost a minute old.”
There’s also a 1.5-second loss of communication between the control processor and the Fieldbus Modules (FBMs) and the Field Device Systems Integrator (FDSI) modules. “Devices hold their outputs at the last value for this time period,” explained Wilde.
“The biggest impact from my end is that all the blocks in the control processor will be initialized,” said Wilde. If they’re set up correctly, PID blocks should notice no impact. Arithmetic blocks will be reset, as well. Your memory points are saved and will be carried through. Your timer functions are saved, as well. The biggest impact is the sequence block. Those fully reinitialize. They will lose all internal memory, and they will be reset back to restart active parameters.”
Other blocks impacted are distributed-control-interface (DCI) blocks. “Each protocol has different fail-safes, so read the manual and test it to see the impact on the PLC,” suggested Wilde.
Best practices
Always read and reread the manual specific to your CP, said Wilde. “As new versions come out, they’re going to be CP-specific,” he explained. “Also, review your PID and arithmetic blocks to understand the impact and the acceptable risk. And review your fail-safe I/O. Don’t forget to review the documentation for the software image to understand what the new features are. Understand what the impact will be the next day and the next month.”
If possible, do an online update with the plant down. “If that’s not an option, prepare plant personnel for possible bumps,” said Wilde. “You’re blind for a very short period of time. I’ve done the update on live plants, but it’s safer with the plant down.”
Lessons learned
Be a good scout. Preparation makes a big difference. “Prepare, prepare, prepare,” said Wilde. “Communicate with your operations and maintenance staff. It’s good to have them there to provide feedback for the next time.”
Set up a testbed, if possible, to look at the impact on your control. “Control loops such as cascade algorithms using peer-to-peer (P2P) should be opened before OLUG,” Wilde suggested. “Additional care should be taken with FDSI communications. It may not be bumpless.”
OLUG requires both fiberoptic cables to be working. “It’s stated in the manual, but it’s easily overlooked,” explained Wilde. “Both A and B channels need to be working.”
Kessler discovered a bonus use case for the OLUG procedure. The plant had a scheduled shutdown, and, when they brought the power back up, there was a frozen peer-to-peer connection. “There were no symptoms initially,” said Wilde. “There was no warning. But operators started complaining about their sequence programs.”
Kessler used System Auditor to generate a report on all P2P connections. “A script was created, which used the ‘omget’ command to compare all sink/source P2P connections to determine the scope of the problem,” explained Wilde. “Once we identified the problem was widespread, we followed the OLUG warm start to reset the whole plant. P2P connections were reestablished with minimal disruption to the plant operation, saving a costly plant outage.”
New FCP280 online upgrade improvements, including bumpless role switch, improved Object Manager reconnection times and no fallback to the checkpoint value, are now in the development and proof-of-concept stage, added Rosborough.
[sidebar id=3]
