Securely Equipping the Remote Virtual Engineer

Joe Feeley is Editor in Chief of Control Design and Industrial Networking magazines. He joined Putman Media in 1997 to help start up Control Design. He has more than 20 years of engineering and management experience in the U.S. and Europe in industries that include high-purity semiconductor products and other specialty materials that require direct involvement with the associated machine designers.

Check out ControlGlobal.com on Google+

The cost of downtime is enormous. "Sometimes we don't have the time to sit back and analyze what it costs, because we're so focused on getting it fixed quickly," noted Sal Conti, Rockwell Automation product manager, introducing his session, "Scalable Secure Remote Access Solutions," this week at the Rockwell Automation TechED conference in Orlando, Florida.

Conti stated that there's $20 billion associated with unscheduled downtime each year. "Some 89% of this downtime is completely random, and a lot of time is spent to resolve these problems," he added. "Eight percent of downtime is spent first trying to determine that there really is a problem, another 21% is spent to analyze the issue, and nearly half of the downtime is spent scrambling to get the needed resources. So three-quarters of the total downtime is spent before the fixing even starts."

Conti recognized that everyone's dealing with leaner staffs, aging workforces that are walking out the door with all that experience, remote locations and employees who are trying to keep up with technology. Further, the response time for a downtime incident can typically take up to 60 minutes. So wouldn't it be great to give your best qualified engineer visibility and access to every site when they're needed?

Three Levels of Security

Conti explained that the Rockwell Automation Virtual Support Engineer (VSE) can provide secure remote access to sites, monitoring equipment and collecting valuable performance analytics. It can help users to better understand how well machinery is working and provide alerts when performance falls outside of predefined perimeters. Because there are different needs and potential threats, and different security rules for different customers, Conti presented good, better and best levels of its Virtual Support Engineer service.

"In a 'good,' or Virtual Support Engineer Standard, approach only outbound communication outside the firewall is allowed, and we require two outbound ports (443 and 80) for that," Conti explained. "We use SSL [secure socket layer] from a tunneling standpoint, as well as user authentication. For users to log into the system, they have to have an active account, and that authentication is brokered by our hosted service center. So there's never a direct connection to the plant. We also have an access audit trail, so we know who logged in and when they logged in."

Conti then explained that the "better" or VSE Enhanced model includes the standard features, but reduces the outbound ports to one (443). It has a couple of levels of certification, including fingerprint, and can limit access by user and/or IP address. It also adds remote access notification, surveillance and recording. "There's also a couple of things added for end-user control, Conti explained. "Not only can the end user allow or disallow access, it can control the type of access and what you can or can't see, the types of IP addresses you can access, as well as some of the recording features."

"There's never a direct connection to the plant." Sal Conti of Rockwell Automation explained how the company's Virtual Support Engineer services allow secure, brokered access to remote industrial assets.

Then there's what Conti labeled the "best" VSE method, which adds compliance with the Rockwell Automation/Cisco Reference Architecture Model to the Enhanced version. "This means you're creating an air gap between the plant floor and the outside world through your industrial DMZ," Coni explained. "There's never any direct connection. It's always brokered by a remote desktop session."

"Just as Virtual Support Engineer allows internal access to remote assets, it can also connect to Rockwell Automation Managed Services, allowing knowledgeable resources to help prevent downtime or optimize your production, in addition to offering support during unexpected failures, all while giving you total visibility and control over who has access, what they have access to and what information they can see," Conti said.

Conti added that the Asset Health Support that comes from Managed Services will report on the health of the network infrastructure devices, the UPS and the server, and the only alarms they get are whether the asset is up or down. It can also monitor the health of medium-voltage drives and PLCs, with service to soon be available for low-voltage drives, MCC, and PlantPAx process automation system.

"The system can also do system or process health for customers, so beyond the assets, we can monitor key process variables," Conti said. "The creation of dashboards and customized data reporting also can be a part of the Managed Services offering."

What the OEM Needs to Know

Along those lines, many industrial machine builders find themselves increasingly involved with providing remote support to their installed customer base. Some might very well use the tools available through VSE solutions, and some might not, so builders need to understand network security and how their machine's communications configurations can impact network security and even their customers' operating continuity.

In an TechED session entitled Security Considerations for Machine Builders, Jessica Forguites, Rockwell Automation product manager for networking and security, outlined her company's approach to security, beginning with the industry- standard defense-in-depth approach.

"Defense in depth protects you against a larger variety of security threats that might be out there," Forguites said. "It also can limit the impact of a security incident. For example, if someone does manage to plug in a laptop with a virus or malware on it, the patch management policy will limit its impact."

Defense in depth consists of:

• Physical security to limit physical access to processing areas, control panels, devices, cabling, the control rooms and other locations to authorized personnel.
• Network security, which includes the network infrastructure, such as firewalls with intrusion detection and intrusion prevention systems (IDS/IPS), and integrated protection of networking equipment such as switches and routers.
• Computer hardening that includes patch management and antivirus software, as well as removal of unused applications, protocols and services.
• Application security, which includes authentication, authorization and audit software.
• Device hardening to handle change management and restrict access.
• Forguites reviewed what Rockwell Automation views as best practices for machine security. "First on the list is to put some sort of security around open ports on the switch, particularly if they're exposed," she said. "Next, change the switch password. Don't use the default setting, and please don't write the password on a sticky note and put it on the switch, as I've seen happen." She also suggests that you limit the number of people who have access to change settings on that switch. Next, Forguites said, limit the size of the broadcast domain by drawing network segmentation boundaries that make sense. Lastly, apply security policies to communications coming into the machine from outside. "You can use access control lists and firewalls to do that," she said.

The Remote Access Decision

Discussing remote access, which is becoming a more-common request of machine users, Forguites noted that even in 2011, 65% of manufacturing facilities allowed some form of remote access into the control system, a number she believes is even higher today. "More recently, in 2013, it was reported that 33% of control system security incidents were initiated via remote access," she added. "So as more companies do this, sometimes out of necessity, you, as the machine builder, don't want to be the weak link in the system."

In the decision-making process for initiating a remote access solution, the risk assessment and the business case need to be determined first, followed by an analysis of risk mitigation techniques and policies. After those elements are established, then the technology choices can be discussed.

"From a business case perspective, there's an evaluation of the cost of downtime, the cost to train and maintain someone locally to handle issues and recognition of the number of assets and functions to be exposed to remote access," Forguites said. She highlighted the safety risk/reward equation, asking if someone unauthorized obtained access, could they potentially do harm and cause a safety risk? Conversely, would a remote access solution eliminate the need to send someone to an unmanned or hazardous location to conduct diagnostics? Other issues to consider, Forguites said, are productivity improvement potential, potential to reduce mean-time-to-repair (MTTR) and evaluation of any confidential information that might be exposed and how one should mitigate that.

For mitigation issues, Forguites said it's vital to develop processes to recognize, react and restore operations back to normal after a security event. In addition, she outlined other sensible steps including wherever possible, moving confidential information to an area not exposed to remote users, limiting the number of assets exposed and limiting access to essential equipment only. In addition, she recommends limiting functions exposed to remote users to essential ones only.

Once you have a handle on these issues, then the technology considerations can begin. "Those include what speed is required, what level of IT involvement and support is needed, what are the clear application needs, what type of auditing capabilities are required, among several other considerations," Forguites explained.

She concluded by saying that these days you generally decide between two approaches to remote access. "You access directly to and through the control system, so the control system environment manages that," she noted. "Or you take an approach that leverages the technologies that the IT group has and maintains and run the access through that segment."