Just because you can't see and touch a computer doesn't mean it's not working, processing data, and crunching numbers as fast and reliably as ever. But this is the psychological hurdle many users must overcome on the way to understanding and benefiting from the drastically reduced hardware and system complexity offered by virtualized computing. And, probably the best way to make this mental leap is to see how others have done it. On the second day of the NovaTech Futureproof Automation 2015 User Conference, Sept. 1 in New Orleans, two major end users showed how they're consolidating computing power and equipment in their plants, and reducing maintenance even as they gain capacity.
Dow virtualizing with VMware and thin clients
John Wyatt, engineer at The Dow Chemical Co.'s plant in Louisville, Ky., reported that it's about to virtualize one of its NovaTech D/3® Systems. It presently runs D/3 version 12.3-2 Distributed Control System (DCS), but will be upgraded to D/3 version 14.1 to manage the plant's acryloid coatings, plastic additives reactor and dryer, monomer distillation and utilities areas. The process applications in each area include:
- Acryloid coatings: one batch solution polymerization reactor train, one continuous solution polymerization reactor train, one extruder and solid-grade packaging lines, a solution-grade drum filling application, and two oxidizers shared by the entire site.
- Plastic additives reactor: two emulsion polymerization reactor trains, three bulk RM storage tanks, two heat exchangers, water treatment and 14 emulsion storage tanks.
- Plastic additives dryer: seven emulsion storage tanks, two spray dryers with 50-pound and FIBC packaging equipment and a pelletizer.
- Monomer distillation: three distillation columns and associated tanks, and monomer loading and unloading equipment for railcars and tanker trucks.
- Utilities: remote monitoring of a chilled water system, and an area for future expansion of the D/3 system.
"Our virtualization will begin in two or three weeks," said Wyatt. "Our hardware was getting long in the tooth and was experiencing a lot of controller and disk control failures, and we'd burned through all the spares. So, it was time to upgrade, but we also decided this might also be the time to virtualize."
Jim Montague is the Executive Editor at Control, Control Design and Industrial Networking magazines. Jim has spent the last 13 years as an editor and brings a wealth of automation and controls knowledge to the position. For the past eight years, Jim worked at Reed Business Information as News Editor for Control Engineering magazine. Jim has a BA in English from Carleton College in Northfield, Minnesota, and lives in Skokie, Illinois.
The plant's upgrade covers 15 display control modules (DCMs) and one configuration DCM (CDCM) to be virtualized into four servers running VMWare, which will be distributed into two clusters with two servers each. Thirteen of the DCMs are used as operator consoles with three to four monitors each. The system also includes 10 operator control module (OCM) workstations in remote locations, and 17 process control module (PCM) controllers.
"The two clusters are in different buildings or areas on site," explained Wyatt. "The CDCM, DCMs and OCMs each run as separate virtual machines (VMs) in one cluster. Each D/3 process has a backup VM defined in the other cluster, and Zerto software keeps backup VMs synchronized with operational VMs."
Additionally, 21 of the desktop operator consoles will be replaced by Wyse thin-client terminals, which will be linked and access their server via Remote Desktop protocol. Also, failover of virtual DCM server processes will be assisted by EMC's Zerto software and a 1-GB network connection between the server clusters.
"Each thin-client machine connects to its VM host via one network connection," added Wyatt. "A second network connection is available, but it requires manual switching and configuration. The thin clients also run under Windows CE-based D/3 keyboards connected via serial or USB links.
Wyatt said the hardware, software and services for the Louisville plant's virtualization and upgrade will cost about $170,000. "This is slightly more than the $150,000 it would have cost to just replace our old hardware, but we know we're going to get a lot more back in maintenance, labor and other savings," he added. "Our costs had looked low due to a surplus of spare equipment, but significant time is required from our automation personnel to replace failing workstations, and there's also the cost of production downtime due to failed workstations."
MillerCoors virtualizes D/3 servers and workstations
Meanwhile, the journey to virtualization at MillerCoors has been going on longer, but it's also reached some major milestones. It's been moving applications to virtual solutions since 2005; it moved D/3 distributed controls at its plant in Eden, N.C., to virtual servers in 2008; and it performed a D/3 upgrade at six other plants from an old virtual environment to a new one in 2014.
"We make beer, which because it's non-lethal and has lower regulatory restrictions, leads to some decisions like not needing redundant PCMs. We can also suffer short outages and plan for recovery, rather than planning to never fail with uptime measured to nine nines," said James Clark, business transformation sub-process owner, MillerCoors. "Our Eden plant was the beta site, so we performed a FAT onsite in our own virtual environment. The goal was to migrate all plants to a similar footprint of virtualization architecture and D/3 versions."
"Our hardware was experiencing a lot of failures, and it was time to upgrade, but we also decided this might also be the time to virtualize." John Wyatt, engineer, Dow Chemical Co.
Though virtualization projects at MillerCoors are getting easier, Clark reported that there was a lot of pushback when they started in 2006. "Virtualization used to seem like alien, crazy talk to the controls guys back then," says Clark. "However, we had 10 new PCMs and a server during 2008, and we were able to play with them and test them, and compare the physical and virtual servers. We even tried to break the virtual server by loading it all while migrating, and it worked fine without a hiccup. That convinced me and a lot of other people."
The six-plant upgrade came about because they hadn't been upgraded in six years, and MillerCoors wanted to move them all to D/3 version 14. "They were all on D/3 version 12, all at different sub-versions and patch levels, and all beyond the three-year contract cycle," explained Clark. "Some plants were in desperate need of new features like online Ethernet configuration. Also, D/3 version 12 wasn't supported on the latest OSs required by our latest IT operating system requirements, and using aging PCs as OCMs put us out of sync with our PC refresh cycle. We had to bring these plants into a more consistent architecture, and so our goal was to upgrade them all at once in one year, and thus be on a consistent and uniform solution for D/3 version 14 and for similar virtualization implementations."
Clark added that the Eden plant's upgrade and virtualization in 2008 consisted of seven virtual DCMs, including one CDCM and one physical DCM as backup, all running Windows 2003 32-bit software. It also included OCMs as remote desktops with Citrix software running on the DCMs along with D/3 version 12.3.
Similarly, the six-plant project in 2014 also consisted of seven virtual DCMs, including one CDCM and one physical DCM as backup, but all running Windows 2008 R2 64-bit standard software. Now, the OCMs are running virtual PCs on Tintri's Virtual Desktop Infrastructure (VDI) along with D/3 version 14.1-2.
"What's new for us in this latest upgrade was moving OCMs to PCs on VMs," explained Clark. "Tintri is the VDI solution MillerCoors is using for PCs running on VMs on thin clients, while D/3 OCM software is installed on a PC image and runs on a VM. As a result, the OCMs running on thin clients are created from one image. We tested that one image until all settings were correct, and then that one image is published and identical for every log in. Consequently, when a thin client boots up, it's always a fresh running image—nothing an operator can do on the thin client is ever saved. This means no problems with anybody changing anything, which is better for security."
Clark reported there are many other benefits of using VM-based OCMs. "The first is that they're easily replaced," he said. "The thin clients can be swapped by configuring only the MAC address on the server, and the already configured settings for a particular workstation will go to the new thin client. This means thin clients and desktop users are same. The very same OCM VM is used by thin clients and non-production desktop users. When an operator workstation thin client boots up, it uses the same image for an OCM as would a process leader logging in from their office. This minimizes configuration time because we only ever need to fix one image. It's like fixing one PC and all the other PCs are fixed.”
Clark added that having a fully staged virtual system that allows for quick re-addressing of nodes allowed for a very fast shutdown of the old system and startup on new system during the MillerCoors upgrade. "Corollary benefits of having a pre-staged virtual system are that we could do extensive pre-testing of applications and pre-staged hardware, such as the OCM thin clients and PCs; we could do multiple system re-builds ahead of time; and using our spare PCM, we could load and identify problems.
"The use of thin clients as virtual PCs running as OCMs is a win. Having a single base image for all OCMs allowed for quick configuration and roll out. Replacement was a matter of unhooking old thin clients or PCs, and plugging in the new one. A dozen OCMs were replaced in a couple hours. Another big win that we often overlook is that D/3 came up and ran, and we had minimal problems. We finished within schedule, and production started that night."
