Where and how does AR, VR fit into process automation operations?

Check out the AR/VR mini series

One area where augmented reality (AR) has gotten its first process industry foothold is in training and maintenance because they're usually non-critical situations where process personnel can try AR and get familiar with it without risking production hitches or possible safety issues.

"Because it's scientifically proven that people retain more knowledge from training if their participation is hands-on, AR/VR-enabled training lets users navigate complex aspects of products by moving virtually around or even inside them," says Rashmi Kasat, vice president and head of digital business development at Metso. "AR/VR can provide powerful immersive experiences. This won't replace classrooms, but it will enhance them."

Enrique Herrera, industry principal for manufacturing at OSIsoft, adds: "One way to get acclimated to AR is training. For example, Fincantieri-Delfi is one of the world’s premier shipbuilders. The Russian and Algerian navies and the Turkish Coast Guard are customers. It's created a platform called ShipView for making complete AR/VR simulations of ships. Complexity is an understatement—the models mesh real-time data streams with documentation on 40,000 spare parts, thousands of technical manuals and other information. With ShipView, they and their customers are replacing classroom training with simulations. Over time, ShipView is becoming a platform for better managing ships in real time. The main difference is that in training, customers are often using historical data and having employees run through curated simulations. Training will pave the way for real-world use."

Remote monitoring—and mentoring

Beyond bringing in and showing production data more efficiently, AR's other main advantage is it can connect field workers with more experienced personnel, who can now see exactly what the field person sees, even though the remote expert is at a distance.

"AR isn't used in many process industry applications yet, but one of its biggest uses is in remote mentoring, mostly in oil and gas facilities," says Bob Meads, CEO of iQagent. "A mentor can see through the field operator's camera, so they're both looking at the same things in the same view. The mentor can draw in the operator's field of view or pull up a document, so they can collaborate on how to correct the issue."

Meads adds that the company’s AR software runs on HoloLens, as well as on iOS devices using ARKit for compatibility with iPads and iPhones. "These applications can help users with maintenance and changeovers, which often require dozens of procedures to be followed," explains Meads. "Instead of using the traditional dozen printouts for changeover, AR lets users pull up the right procedure on a wearable device, including images and videos, make that information available in context, and walk users through what they need to do."

Russ Agrusa, president and CEO of Iconics, now part of Mitsubishi Electric Corp., adds that, "Remote experts can be whisperers that solve problems, but AR can also connect the few remaining process control gurus with new people that can learn from them more easily. AR improves collaboration because those process control gurus can write notes, point out parts of images, and circle items to convey their expertise to new employees in real time. Iconics has a Connected Field Service solution that integrates with our MobileHMI app and remote expert feature, which connects regular SCADA systems; allows them to run on smart phones, headsets or eyepieces; brings up any related information (from documentation to videos); and fuses them all together."

Wearable = hands-free

Probably the most important technology aiding AR and enabled by it is the parallel emergence of wearable interfaces that can put data in front of workers, deliver remote expertise to users, and most importantly, free their hands at the same time.    

For instance, Braskem Idesa makes more than 1 million tons of polyethylene annually at its three-year-old plant in Nanchital, Mexico. Its products are made from ethane using a potentially risky process involving pressures up to 3,000 (45,000 psi). To improve performance and reduce risk using digitalization, machine learning, and predictive/prescriptive maintenance, Braskem personnel undertook a project called Cyclops that uses wearable, AR-enabled, Connected Plant headsets from Honeywell for two-way sound and video communications. They're supported by a three-year subscription for software, applications and services for Connected Plant wearables. The AR headsets provide task automation to workers, who receive guidance and can visualize system information and documents to correctly perform tasks. They can also get assistance from experts, who can access each user's head-mounted display camera and use video chat to offer advice.

"Cyclops began after I talked with Honeywell about scenarios as I worked with the plant on startup in 2016,” says Marco Santos, production engineer at Braskem Idesa. “My concerns were focused on the operators. We had cell phones, laptops and the DCS, but in the field, no tools for operators in the field, just pencil and paper to take data.

"With Cyclops, we’re increasing reliability, productivity and operational skills, and monitoring startup and shutdown activities,” adds Santos. “We can use the headsets to communicate and coach operators in real time. When we inspect raw materials, we can consult specifications. For safety, operators can take video evidence of unsafe acts and conditions, and report them in real time to other operators and supervisors."

In addition, the headsets also let experts guide staff using text and graphics, designate features that need attention, capture and annotate still images, and make text notes. Also, videos of field operations can be used as tutorials to teach new operators about unfamiliar procedures. In the future, Santos adds the plant will implement Movilizer, which is a handheld system that guides and records operator and maintenance rounds, and uses intelligent vests for real-time monitoring of workers' vital signs and environmental conditions.

"AR is the overlay of digital information on the physical world, using smart glasses or a mobile device," says Vincent Higgins, general manager of the Connected Plant program at Honeywell Process Solutions (HPS). "AR is used in the field to help workers be more effective and safe. Different from this, a digital twin is a virtual model of a process, product or service. The digital twin brings together data from the physical world, often comparing it to a model, and allowing analysis of that data for monitoring of systems to predict and solve problems.

"Consequently, we've brought together our Operator Training Simulator (OTS) that usually trains control room operators based on physical simulators and our Immersive Competency (IC) VR software platform. This allows hands-on training of control room and field personnel at the same time in a highly realistic VR environment, communicating with each other in a virtual world environment. Many process industry companies are being directed to transform with digitalization and data analytics, and XR is part of that answer. We partner with companies to help them in their digital transformation, working with them closely, looking for use cases, piloting solutions, and then scaling up."    

At the same time, HPS has also launched its Intelligent Wearables platform, a combination of voice-enabled heads up display connected to a hard hat, running applications that transform the way the field operator works, with instant access to real-time data, documents, video and analytics that provide new insights. "Shell has already standardized on Intelligent Wearables in 12 countries, and 24 operating sites worldwide," adds Higgins. "As we go forward, AR can also bring in other technologies that already exist, such as geolocation and geofencing for improving alerts and awareness, and biometrics for monitoring worker health."

Higgins reports the four main applications for Intelligent Wearables can also be used on tablet PC and smart phones are:

• Expert on Call video chat, where the fieldworker shares the wearable's front-facing camera and audio, so office based experts can provide immediate guidance and advice, remotely controlling the camera, zoom in on plant-floor equipment, and circle components on screen.
• Video Assist that creates a repository of user-created videos that can be searched and called up with voice commands, which further helps with skills gaps and knowledge transfer. It will let experts record comments about what they're doing, and create an instructional library for less-experienced colleagues;
• Document Assist that connects to a plant or organization's document management system, so users can access P&IDs, schematics, lists and other materials, also using voice commands. This application delivers its documents to wearables or other interfaces.
• Honeywell Forge Inspection Rounds allows field users to follow digital work procedures, connected into SAP or other asset management systems, reducing their reliance on paper-based procedures. When users find an anomaly on the plant floor, they can take photos and capture other data, then immediately kick off a digital work order to begin repair, along with tracking progress and user location, gauging user efficiency and safety, providing visibility to things like  mean time to repair (MTTR) and overall equipment efficiency (OEE).

Guided to performance

Despite the potential gains, Dr. DP Prakash, global head of innovation in the CIO division at GlobalFoundries in Santa Clara, Calif., which recently implemented these technologies, reports that it took a while to convince staffers to accept and get used to their new AR tools. "It's human to resist change, so we identified pain points, and showed how AR could help," explains Prakash. "We made it more about the business and use cases, and less about the new technology, and got a much better reception from users. For example, once they learned how AR could help their standard operating procedures, they came knocking and asking for it."

For users that want to implement AR, Prakash adds advocacy should come from the top, bottom and middle of any organization, just as it did at GlobalFoundries. "Leading a business transformation through AR is not an easy task, and it will be much tougher for any organization without support from the CEO, CIO and other top managers. You have to get engagement at the top floor early on. Our CEO and CIO recognized the potential of enterprise AR to provide a quick return on investment (ROI) that led from the front," says Prakash. "Leadership on the plant-floor is just as important because it has the people who are closest to the pain points and know the most about them. They just need to learn what AR can do and how it can deliver ROI across the four levels of value, including documentation, training, operations and analytics.

"Finally, middle managers, who usually focus on key performance indicators (KPI) and other metrics, and don't have time for innovation need to be convinced. Our onsite Innovation Labs play a crucial role in this regard, where use cases with benefits can be demonstrated leaders without disrupting operations. All GlobalFoundries' sites now have fully enabled labs, where new ideas are put to test with a culture of risk taking, failing fast and learning quickly."

Suellen Lorkowski, senior technical consultant for information solutions at Rockwell Automation, adds that, "If you have to assemble a complex system or ramp up the skills of some new employees, AR can be very helpful in walking through what to do and how to do it correctly, especially if they can use it on wearable devices like Vuzix glasses and earpieces, HoloLens headsets or RealWear devices. In our case, wearables can run FactoryTalk Innovation Suite software to show users how to do their work, and display digital images over physical images. This can help workers reach targets with less frustration, which improves job satisfaction and retention."      

Augmenting analytics

Beyond its initial applications in training and maintenance, AR can help in other disciplines, such as keeping data close at hand, models updated and analytics applicable.

"A couple of years ago, I'd have said, aside from the usual use cases around training and access to documentation, AR/VR was window dressing, but my position changed. AR can also be a major source of context and reinforce domain knowledge in analytic models. This also allows organizations to move forward with more consistent asset models" says Luke Durcan, director for EcoStruxure for North America at Schneider Electric. "Many clients talk about wanting to use digital twins, but they don't really know what it means or where to start. I ask what asset register they're using such as Wonderware, Maximo, SAP or manual documentation in a file cabinet because they can turn this registry into the start of an asset model once they've enriched the data, and progress from there to a potential digital twin. However, these registers and models are usually fragmented across each organization, and they need to be more consistent to put data into context, so they can run higher performing analytics."

Figure 1: The 16 robotic cells at Schneider Electric's 50-year-old but newly revamped smart factory in Lexington, Ky., which makes about 11,000 load sensors per day using its EcoStruxure Augmented Operator Advisor (AOA) software to increase visibility into operations maintenance, achieved a 20% reduction in mean time to repair (MTTR) on equipment, and use process digitization to reduce paperwork by 90%. Source: Schneider Electric

Durcan reports users need consistent asset models that can correspond with time series data, which is the foundation of any data science or machine learning program. "However, data science is a new concept for many users, so they need consistent models and asset registers from which they can build digital twins," explains Durcan. "This is where AR can come in as the interface between plant floor and the digital infrastructure. Traditionally, an asset register might just be a document in a binder, database or online, but these can get lost, and if they're not continually updated, they'll lose their context, even if they're using software like Aveva APM, Maximo or SAP. It's not just the technology; it’s the people and process that drive business value."

In short, AR/VR can give users the nudge they need to update their asset registers and models, as well as provide a consistent framework for adjusting applications and enabling patchworks of equipment to collaborate. "If a maintenance guy must keep his operation running, documentation is likely low on his agenda. However, if he can use an AR tool, such as our EcoStruxure Augmented Operator Advisor (AOA) software, then he might be able to do more documentation because it's easier. Moving forward, high-performance models are getting more complex, and using direct, ground-truth data feeds from devices back to models, and AR can enable them by adding context, or allowing domain experts to better train data science models."

For example, Schneider Electric runs 16 robotic cells at its 50-year-old brownfield facility Lexington, Ky., which makes about 11,000 load centers per day for residential and industrial use and industrial safety switches. However, like any manufacturing environment, machine breakdown could effect production especially as much of the plant is automated. As a result, Schneider Electric recently revamped the plant as its first U.S. smart factory by integrating EcoStruxure AOA, which increased visibility into operations maintenance, achieved a 20% reduction in mean time to repair (MTTR) on equipment, and used process digitization to reduce paperwork by 90% (Figure 1).

"The Lexington plant is using AR for training, compliance, and speeding up documentation for diagnostics," adds Durcan. "Unless it's part of a mandated process, a lot of data doesn't get saved, or if it's kept, it's only for an individual process and it's locked up. AR can change this by improving data access for asset models and operations."

In addition, other gains at the smart plant include:

• EcoStruxure Resource Advisor and Power Monitoring Expert software achieve 3.5% YOY energy savings and $6.6 million in regional savings since 2012, while their reporting capabilities and increased transparency drive operational performance;
• Aveva Indusoft Web Studio software delivers edge digitalization of paper processes to eliminate paperwork by 90%, while cloud connectivity enables digital dashboarding;
• RFID OsiSense solution eliminates 128 daily fork truck miles, and eliminates $500,000 in work in progress (WIP) inventory with a 33% first year ROI;
• Aveva Insight Data software unlocks and shares silos of data in a mobile manner, reducing downtime in processes by 5% with ROI of less than six months; and
• Magelis GTU/GTUX HMI provides agile operator management of the processes and vivid visuals of it onsite and via mobile devices.

"Because it can keep data handy, AR can also reduce downtime by allowing users to tweak their models easily," says Rockwell's Lorkowski. "I always ask 'is this a living document or not?' because I grew up in automotive plants where things were constantly changing. AR pushes this to the forefront because it allows quicker updates, so workers on the line know they have the latest instructions and animation to build products or do other procedures. Also, AR can help ramp up workers on new procedures before they go into full production.

"The best way to implement AR is to see how it applies to your own work and operations, and decide what resonates. One company may get the most value from training workers, while another might not, so it's important bring together a lot of people in the organization to see where AR would be useful and still have the biggest impact."

Kim Fenrich, digital services product manager in the Industrial Automation business at ABB, reports: "AR takes the data dashboard to another level by making data more meaningful and displaying it in relation to the equipment where it was produced. This means there's less need to ask or show what that data means, and potentially less need for the 30-40 years of experience of the guy that just retired. AR also improves cognitive learning, and allows users to better remember their experience with a process due to the visual format it’s presented in."

Fenrich adds that ABB Ability digital solutions are available on tablet PCs, smartphones or RealWear devices, or integrated within products as part of their service contracts. "Today’s ultimate goal is the digital twin, which provides a crystal ball of a piece of equipment, process or plant. These twins will be integral to how plants and processes are run, and our customers will expect to engage and do business this way," says Fenrich. "AR can help provide almost instant access to device status, historical data and simulations. This will also connect asset management to deeper simulations and AI as part of the overall drive to more autonomous operations. The result will be more empowered operations, maintenance and engineering personnel, so they won't need to call or rely on as many other people."

Seeing the future

Once AR, VR and mixed reality become more mainstream, they're expected to surpass merely presenting data to altering how interfaces and even some controls are deployed, but it's suggested that users and organizations proceed cautiously at first.    

"With AR it's important to start slow," says OSIsoft's Herrera. "There are two big hurdles to overcome. The first is skepticism. Many employees think AR might make them sick or interfere with their ability to do their job. A lot of executives believe it will be just another costly experiment that goes nowhere. It’s hard to forget the Google 'Glassholes.' The second hurdle is technology. AR developers will have to come up with ways to serve up information people will need in a way that doesn’t become a distraction. The user interfaces and data simulations need to be improved before AR can become widespread. As a result, the best thing companies can do right now is experiment—find use cases where AR provides a distinct advantage, test the usability and usefulness, get feedback. At some point, we’ll see the killer use cases emerge: that’s when the technology will start to take off."

iQagent's Meads adds, "AR and wearables are going to be the way to visualize data from PLCs, cloud-computing services and IIoT devices. They'll gradually replace tablet PCs and smart phones, too. Plus, many SCADA screens that used to be difficult to create for remote viewers will become more intuitive because users will be viewing AR resources in a relevant context, and they won't need as many complex screens as they did before."