Along with patchwork programming, outdated servers and process invisibility, Land O'Lakes’ cheese and dairy proteins production facility in Melrose, Minn. was faced in 2015 with imminent obsolescence of the controls it relies on for production.
“Our PLC-5s were nearly 100% full, slow and outdated. We were using physical servers on the plant floor, which were slow, outdated and running Windows Server 2003. Software was not updated. Components were taking two weeks to find, and we had too much downtime,” said Nikolas Paris, maintenance manager, Land O’Lakes, during his session, “Land O’Lakes Modernizes with Complete Code Rewrite, Minimal Downtime,” co-presented with Mark Visness, engineering manager at control systems integrator Cybertrol Engineering, Minneapolis, at Rockwell Automation TechED, this week in Orlando, Fla.
A previous upgrade by a different system integrator had failed, mostly because of a “network incompatibility that was beyond their scope,” Paris said. “Code had been developed by various vendors with entirely different structures. We couldn’t troubleshoot or add anything – no one could read the code.
“We needed process data and visualization to bring us into the current millennium, not reading gauges on the plant floor and entering data on a clipboard.”
Refreshing the hardware
Replacing the existing control and information systems began in 2015, when the plant upgraded from physical to virtual servers, and updated the network infrastructure, operating system and software. Additions were also made to improve visibility and support.
[sidebar id =1]
A new VMware virtual server infrastructure gave high availability with shared storage and scheduled backups. “Virtualization lets you run multiple virtual computers on a single physical machine, with multiple operating systems and different software,” said Visness. Redundancy and failover capabilities give high availability, and users can move a virtual machine (VM) to a second server to allow updates while running. Snapshots can be scheduled, for example, every 15 minutes for backup. In case of a failure, you can switch to the backup and be back online in minutes.
Different virtual servers handle different functions. For example, FactoryTalk Directory, FactoryTalk View SE HMI, RSLinx Enterprise I/O server, FactoryTalk View client terminal server, FactoryTalk Historian SE, FactoryTalk Batch, FactoryTalk VantagePoint EMI reporting, domain controllers, engineering servers and backup servers each run on its own virtual server. “In this case, we have 14 virtual servers running on two physical machines,” Visness said.
In 2016, the plant made the move from Allen-Bradley PLC-5 controllers to Allen-Bradley ControlLogix control systems. RSView32 HMI was replaced by FactoryTalk View SE, with a complete code rewrite and additions to improve reporting and information visibility. The change to the ControlLogix control systems had to be done in less than 48 hours. “Cows keep giving milk,” said Visness. “A longer shutdown would require diverting milk to other plants. We held the shutdown to 32 hours.”
Keys to the rapid migration included:
- Using conversion modules to convert 1771 I/O to ControlLogix I/O modules on the 14 PLC-5 racks. “We can change out an entire rack with no rewiring. We started at 6 a.m. and completed the hardware conversion by noon,” Visness said.
- Converting the PLC-5 code as-is, to verify operation on the new processor.
- Spot-checking the I/O
The next 24 hours was used to run the plant on water to verify production operations. The plant was up and running again in 32 hours.
Commissioning new code
Writing and commissioning the new code took longer, and was done by unit operation: milk receiving, pasteurization, ultrafiltration, evaporation, etc. The existing code was undocumented. Much of it was bad, obsolete and/or irrelevant to current equipment and operations, in short, “We had difficult-to-edit code for equipment that didn’t exist anymore,” Visness said.
The process for writing new code included extensive observation of operations and discussion with operators, Visness added. “We’d ask about some code and they’d tell us that acid reclaim tank is now a caustic reclaim tank, and we reclaim acid in that old milk silo over there.”
Bringing in the operators early was crucial to the project’s success. Engaging the operators in writing the new code and training them on the new system ahead of time “pared down the punch list and streamlined the commissioning process,” Paris said.
For each unit operation, all-new PLC/HMI code was written and installed alongside the existing code, but not activated immediately. At the end of a daily production or clean-in-place (CIP) run, new code was activated and commissioned using toggle bytes to increment the activation. “We activated new screens as we reached them,” Visness said.
A unit was upgraded every two to four weeks. “After the initial shutdown for hardware replacement, there were no further interruptions to production,” Visness said. “Operators had time to adjust to the new screens and other operational changes.”
The hardware cutover was done at the end of March 2016, and the software migration was completed in January 2017. “This approach allowed us to upgrade and recommission a system with nearly 1,000 I/O points and several different unit operations in a systematic way,” said Visness. “We were able to do this with a smaller commissioning team. What might require 12 people was done with a team of four – two on day shift, two on night shift.”
Upgraded performance
Along with solving problems posed by obsolescence, parts availability, and support, the upgrades have made a material improvement to operations by reducing downtime and making information visible. For example, CIP program documentation was not up to date, with many changes over the years, dead code and no reporting beyond a circular chart at the end of the day. A new recipe-based CIP system now uses downloadable sequence charts for control, with CIP steps easy to view and edit by the sanitation manager, and reports show real-time plus event-based data.
“Users can change steps and equipment, set start and end times, review all the CIP procedures and reports,” Paris said. They can see who ran it, and see temperature, flow and conductivity trends, time-stamped CIP steps, and summaries including exceptions, alarms, events and operator comments.
Future plans include taking advantage of increased visibility of production and energy information. For example, cheese powder dryers are major energy consumers. “Operators now have access to data on uptime/downtime, pounds per hour vs. target, and moisture level throughout the day as they run the shift,” Paris said. “Information on natural gas usage, water flow and usage, efficiency and runtime are reported automatically at the end of each shift. We can correlate that, identify the causes of anomalies, and improve our performance.”
