Oh sure, Ethernet is just a cable and some software, but pretty much anything and everything can happen on it—and happen fast—and that's its main attraction.
At its most basic, Ethernet connects computing devices, establishes local area networks (LANs), virtual LANs (VLANs) and wireless LANs (WLANs), and runs multiple digital communication protocols, mostly those based on Transmission Control Protocol/Internet Protocol (TCP/IP). Of course, this makes Ethernet the backbone of most modern, mainstream, IT-based, wired and wireless networking, which underpins most if not all digital and Internet protocol (IP) communications, virtualized computing, cloud and software services, the Internet of Things (IoT) and the Industrial IoT (IIoT).
"Industrial Ethernet—and EtherNet/IP in particular—has proven itself to be a consolidating force for automation across discrete, hybrid and process industries," says Katherine Voss, president and executive director of ODVA. "Going forward, EtherNet/IP and complementary technologies will continue to evolve in step with the IoT, so users in the process industries can deploy industrial Ethernet in more and more field devices."
Jessica Forguites, networks product manager and platform lead for network infrastructure at Rockwell Automation, adds, "This is an exciting time for networks because IoT is transforming them. So, where we used to have multiple servers and networks, we can now do all the same functions on the same technology, and share more resources more easily across multiple virtual servers. However, we also must be more careful in planning and coordinating these networks, and invest in the Ethernet backbone they'll need."
Advantages spur adoption
However, despite its historical dominance in IT and mainstream realms, Ethernet has emerged more slowly in industry and manufacturing, especially among cautious and safety-focused process engineers and their applications. Fits and starts along the way have been common.
"We still have many dinosaurs in the process industries that aren't willing to move from 4-20 mA and relays, and so we still have a lot of hardwiring where there should be networking," says Carl Henning, deputy director of PI North America. "This is one of the reasons why we still do so much education, and clearly, as more young people come into the process control field, more networks will get updated."
Chris Hamilton, technical consultant for manufacturing in operations technology (OT) and IT at Grantek Systems Integration Ltd., a CSIA-certified system integrator with seven offices in North America, adds that, "We have a lot of clients with significant network loads and increasing responsibilities, who may have an Ethernet reference architecture document from a supplier, but have limited capacity and experience in how to access and then actually execute against the changes required. Also, because so many networks are evolving from traditional two-wire to Ethernet, many grow by just adding ports in totally unplanned ways. Grantek's expertise and our consulting process enable us to work with our clients, evaluate their current sites, understand their current and future business needs, and then develop migration plans and roadmaps that advise them to address their highest-risk equipment and systems first."
Nonetheless, Ethernet's overall momentum and increasingly proven capabilities have allowed it to inch ever further down onto plant floors, device levels and field components—and move from monitoring some process applications to even actuating changes in some of them. "Over time, we're going to see open network protocols and technologies push onto the device level, and they'll be different than supplier-developed versions," says Matt Newton, technical marketing director at Opto 22. "These open protocols and technologies will be more web- and IT-based like HTTP [hypertext transfer protocol] or RESTful APIs that uses HTTP requests. These well-tested, open and standards-based IT tools pushing into manufacturing are a great opportunity." RESTful APIs is short for "representational state transfer" and "application program interfaces."
For instance, Frakes Engineering, a CSIA-member system integrator in Indianapolis, performs water/wastewater jobs all over the Midwest, mostly in Kentucky, Ohio, Indiana and Illinois, and reports that fieldbus and Ethernet projects are multiplying. "About two years ago, we helped Lafayette, Indiana, upgrade communications with its remote wells and water towers," says Mike Anthony, senior engineer at Frakes. "They'd been using 1,200-baud (bps) radio/telephones, probably talking Modbus, and we replaced them with about 20 of Cisco's IE 3000 managed Ethernet switches, eight CompactLogix PLCs and two redundant ControlLogix PLCs, all communicating via EtherNet/IP, and able to link to a fiber-optic network the city recently installed at its fire stations. When we switched to the new network, the operators and engineers were really happy because they went from 1,200 bps and waiting minutes for data to 100 Mbps and getting everything instantly."
Darin Heitman, network engineer at Frakes, stresses it's especially important to use managed Ethernet switches because they can determine network topology, establish VLANs, segment networks, regulate data traffic, prevent feedback loops and broadcast storms, enable configuration to mitigate risks, control and shutdown network sections for security purposes, and show what devices are connected where. "Managed switches and a VLAN let you get on your network, and they indicate where a problem is," says Heitman. "Unmanaged switches might look OK, but they can stop, and they don't show where to look when you're troubleshooting. We're able to participate in Lafayette's fiber-optic network, but we can also pull only the fiber pairs we need, and maintain isolation where it's required."
Interoperability and chips
Figure 2: Nisse Olsson, operations manager at Järvsö ski center, uses a regular PC to monitor and control two Internet-enabled Emotron VSDs at a pumping station 3 km away via Modbus TCP/IP and HTTP. If needed, support technicians can access all parameters in the VSDs remotely via the Internet and a VPN tunnel. Credit: Emotron
Because it appears to be everywhere, it might seem like Ethernet can't be stopped, but there have been hurdles over the years that still hamper its progress. The first is that Ethernet and its IEEE 802.3 standard mostly encompass only its physical media, so everyone has been able to use it even if none of their devices can actually talk to each other. Luckily, increasingly fast, powerful and inexpensive microprocessor chips have been allowing many developers to translate data and move it between the major serial and digital communications protocols running on Ethernet, enabling them achieve interoperability, or at least interoperability with a little added latency.
"The industrial Ethernet protocols reflect the philosophies of their parent companies, but those big players don't always think alike," says Phil Marshall, CEO of Hilscher North America. "So, we have different network architectures that have to be implemented consistently across thousands of products reflecting different control philosophies. But what plant managers want is to produce more widgets—not worry about different networks."
Marshall adds that Hilscher bridges the interoperability gap between various suppliers' equipment with gateways built on the company's netX chip family of seven—and soon eight—microprocessors that enable components running the different protocols to communicate, exchange information, and even perform control functions. "And, with netX at their core, our gateways can be used to convert or upgrade any fieldbus to network to any industrial Ethernet."
Ariana Drivdahl, product marketing manager for industrial Ethernet at Moxa, agrees that microprocessor-fueled switches can natively support different Ethernet protocols, which means they don't have to be programmed like compliant end devices, and pass protocols seamlessly in real time with little backlog. "Previously, users had to know everything their network had to do and was going to do, and then do the timing and programming," says Drivdahl. "Now, they just tell the switch what they want to do. I think the biggest leap that allowed this to happen was the emergence four or five years ago of the major Ethernet protocols like Profinet, EtherNet/IP and Modbus TCP/IP, which allowed us to build functions of these protocols into our switches without having to add additional chipsets and costs."
Down on floors, out in fields
Besides simplifying networks and saving on cable, hardware, labor and maintenance, Ethernet's other main benefit is it lets users reach deep down and/or faraway devices that were probably ignored before, and secure new signals and their valuable information.
For example, after building two mines at the Chichester Hub and Herb Elliot port-and-rail facility in Port Hedland, Australia, Fortescue Metals Group Ltd. (FMG) recently started building a new iron ore mine at the Solomon Hub (watch the video below), located 120 km west of Chichester in Australia's Pilbara region (Figure 1). The first two mines used Profibus for controls and electrical networking, and thanks to its reliability in their extreme temperatures and noise, FMG decided to try Profinet for Solomon's PLC-to-remote I/O communications; future-proofing and coordinating with its Proficy Plant Systems (PPS) from GE's Automation & Controls.
Expected to produce 60 million tons of ore per year, Solomon Hub includes two iron ore mines, three primary and secondary crushers, two ore processing facilities, 15 km of overland conveyors, and a two-canyon stockyard for shipping ore to Port Hedland. From crushing pads to train load-out (TLO), these facilities include 15,000 I/O points controlled by 200 Profinet devices, 40 PLCs, 50 GE Cimplicity HMI workstations and 15 servers. Their implementation was designed by system integrator I&E Systems in Perth, Western Australia, using its DAD System Information Modeling software.
Figure 3: While standard Ethernet occupies the enterprise, demilitarized zone (DMZ) and control domain levels, industrial Ethernet protocols such as ODVA's CIP-based EtherNet/IP are connecting to many devices on plant floors and process applications via gateways, switches, I/O modules and other translating devices to access fieldbuses, wireless, 4-20 mA signals and their operating data. Credit: ODVA
By using a ring topology on each PLC, I&E reports it could easily isolate devices for increased commissioning and maintenance flexibility. "We commissioned a sample station that had its marshalling panel in the middle of a ring of marshalling panels for a conveyor," says Tom McCarthy, I&E's lead engineer. "We could power down the marshalling panel, but the conveyor would stay running due to the dual paths of the ring." To complete the project, Profibus handles inter-PLC control of Solomon's processing infrastructure.
Likewise, Vinterteknik Sundsvall AB in Sundsvall, Sweden, manufactures snow cannons, pumps controlled by variable-speed drives (VSDs) and irrigation systems for clients such as the Järvsö ski center. Järvsö recently needed to supply water to its cannons from a river and pumping station located 3 km away over difficult terrain, but control the pumps from the center. The station has two FDU VSDs from Emotron that control two 250-kW motors and drive two pumps. Three more pumps at the control station boost pressure up to the cannons, and they're controlled by one FDU and two Emotron MSF soft starters.
To avoid costly, dedicated cabling and likely signal attenuation, Emotron recommended secure, remote access via the Internet, which was possible because its VSDs can communicate over industrial Ethernet. Because IP is non-deterministic, Emotion reports it uses the Internet mainly for logging and configuration, though it can also be used to control non-time-critical applications.
The VSDs are equipped with Anybus CompactCom plug-in modules from HMS Industrial Networks, which communicate via Modbus TCP/IP and HTTP. This allows an integrated web server using HTTP to extract and present parameter information from the VSDs, and provide access to connected units via a web browser. Consequently, monitoring and control can be carried out from a PC with a VPN client without added software (Figure 2). To improve security, most communications between the pump station and ski center are on a regular PTSN telephone network via ADSL, but their DR-250 router from Westermo can automatically switch to wireless 3G as a backup.
Figure 4: John Heidelberg, network administrator at Jensen International, uses LinkRunner AT network auto-tester from Fluke Networks to speed up testing of his company's Ethernet networks and quickly gauge connection continuity, link/speed/duplex, name and IP addresses of nearest switches, server availability, Power over Ethernet performance and resource connectivity. Credit: Fluke Networks
"This solution allows remote configuration and fast, accurate control," says André Rowéus, technical specialist for communication solutions at Emotron. "It enables systems to exchange large amounts of information quickly, while also providing a user-friendly interface. User can analyze collected data, optimize processes, eliminate bottlenecks, locate and correct faults faster, and maintain a redundant network topology that improves safety."
Before achieving gains like these, however, it's crucial to know the applications and its requirements, according Jeff Lund, senior director of product line management for IoT at Belden. "You first need a solid understand of what you've got, and then you can determine what you need," says Lund. "So, what hardware, protocol, temperature, vibration, approvals, security, environment, wireless, form factor and other requirements do you have? It also resonates with users that it will likely be important to get double-pull tensile strength on cable. This is because, when you ask a plant manager how many hours their staff spends troubleshooting to find bad cable, their answer will most likely be measured in days."
Guided by wireless
Not surprisingly, after Ethernet freed its initial users from so much point-to-point hardwiring, it's only natural it would help them and their applications shed the bonds of gravity, too—by going wireless with Wi-Fi and its IEEE 802.11 standard. Richard Wood, product marketing manager for network infrastructure at Moxa, reports that wireless Ethernet has some unique challenges that go along with its benefits. "Wireless Ethernet is available anywhere, can be deployed anywhere, can serve in moving applications, and secure previously unreachable signals," says Wood. "But at the same time, it's susceptible to EMF and other interference, and requires users and integrators to know how to do a site survey before deploying it. So, even though wireless has grown by leaps and bounds, there's still a lot of reluctance about using it."
For instance, Korea Electric Power Co.'s (KEPCO) coal-fired plant in Hadong consumes huge amounts of coal to generate about 6% of the nation's electricity, and performs extensive coal testing and analyses to optimize its efficiency. Facing the expense of laying 600 meters of fiber-optic cable a few years ago, KEPCO and its distributor, Ajin Systech, opted to go wireless, and adopted RadioLinx industrial hotspot radios from ProSoft Technology, which run at up to 54 Mbps, using IEEE 802.11a in the 5-GHz band.
In the plant's testing and analysis process, a single ControlLogix PLC is connected to a Flex I/O system at remote coal-handling stations. The RadioLinx radios are installed on the conveyor tower and on the master ControlLogix PLC, which allows input and output data to be consistently and securely transferred between the two radios. This lets Hadong's engineers now at any minute whether their bucket-and-conveyor system is transferring coal properly.
Similarly, to modulate wind power generation and help older farms comply with California's energy rules, SCADA Solutions in Irvine, Calif., has developed its WindCapture online monitoring and control system. It includes SNAP PAC industrial controllers from Opto 22, which supervise individual I/O controllers on each turbine, communicate via a wireless radio network such as Wi-Fi, and use Opto 22's groov operator interface for mobile devices. This combination lets WindCapture manage wind production down to individual turbines, and quickly scale energy production up or down based on the grid's real-time energy requirements.
"We can take analog and digital data from the field at the network edge, and bring it into control software that monitors and manages the whole wind farm," says Craig VanWagner, engineer at SCADA Solutions. "Opto 22's software gives us a variety of interfaces and protocols and the ability to push OPC data to the cloud, while groov enables our users to access live turbine data, and control them from a smartphone or other mobile device from anywhere in the world, in near real time."
There are many ways to bring Ethernet into process automation and control applications, and do it successfully, safely and securely. Here are some of the most helpful:
- Use shielded cables and connectors to minimize electrical noise, jitter and other kinds of interference;
- Make sure control panels, server racks and other network infrastructure equipment are in physically secure locations;
- Separate process control network from IT, business-level and external networks by using managed Ethernet switches as firewalls;
- Segment control-level network into functional subnets with managed switches applied as firewalls;
- Use managed Ethernet switchers to establish virtual local area networks (VLANs), and use VLANs to prioritize network traffic to avoid data collisions and dropped communications;
- Don't keep plugging Ethernet cables and devices into available ports daisy chain-style—plan and carryout a specific strategy for your entire network;
- Inventory, audit and evaluate all existing networks, components, ports, wireless connections and other access points in your network;
- Conduct a risk assessment of your network, determine what connectivity data capacity you truly need, and draft a roadmap to achieve them;
- Establish a patching policy and procedure for incoming software updates;
- Investigate and apply appropriate, IT-based network traffic examination tools, and set up available alerts; and
- Continue to monitor evolving probes, intrusions, threats and attacks, and address with updated solutions.
IT tools to the rescue
Even though industrial Ethernet can save on hardware and provide better data, many potential users still don't have the manpower, funding, expertise or training to upgrade to it. Fortunately, many IT-based tools are continuing to get easier to use, less costly and more approachable for more users. Lund reports that Belden's HiVision software will show users all the devices and IP addresses in their network, and draw an onscreen graphic of them.
Frakes' Anthony adds that water/wastewater departments in many smaller communities face tougher challenges modernizing and migrating to Ethernet because their staffs are smaller, their equipment is older, and they often have little if any IT support.
"A rural water system many only have three guys working days, no one at night, and a 10-year-old box PC running their plant that no one has touched since it was installed," adds Anthony. "I was just at a meeting in a small town that's switching from an old Rockwell Automation and Windows NT system running their water plant and distribution system to several Stratix 8000 managed Ethernet switches using EtherNet/IP to reach two servers and four virtual workstations running Microsoft Server 2012 software, hypervisor software from VMware and Rockwell Automation's FactoryTalk View software. This is just one plant that has eight PLCs for filtering and eight more PLCs for handling chlorine, fluoridation and high-service pumps."
Since it was invented by Robert Metcalfe and David Boggs at Xerox PARC in 1973, launched in its first commercial version in 1980 and approved as the IEEE 802.3 standard in 1983, Ethernet has steadily gained capacity and bandwidth from its initial 2.94 megabits per second (Mbps) to 100 Mbps, 1 gigabit per second (Gbps), 10 Gbps, 100 Gbps, and soon 400 Gbps, scheduled for introduction in 2017.
To travel farther, Ethernet typically switches to fiber-optic cable, which can go up to 2 km at 100 Mbps in multi-mode or up to 10 km at 100 Mbps in single-mode. Higher data rates mean shorter distances, and so 1 Gbps can go only 1 km in multi-mode, and 10 Gbps can go only 550 meters in multi-mode. To go airborne via a WLAN, Ethernet moves to equally well-known Wi-Fi, which is the IEEE 802.11 standard that uses mostly 2.4 GHz and 5 GHZ frequencies in the super-high-frequency (SHF) and industrial-scientific-medical (ISM) radio bands.
Once these avenues are open, digital data can move over Ethernet in data packets that transport an Ethernet frame and its data payload, which is specified by the IEEE 802.3-2012 standard. This frame's structure consists of:
- Seven-octet preamble (each octet is eight bits) and one-octet start-of-frame delimiter (SFD) that address the physical layer;
- Ethernet header with six octets each for a media access control (MAC) address destination and MAC address source;
- Two octets to designate Ethernet type, either IEEE 802.3 or Ethernet II;
- 46 to 1,500 octets of payload data; and
- Four-octet frame check sequence (FCS), which is a cyclic redundancy check (CRC), to end the frame.
Beyond defining its basic data coinage, Ethernet is also organized in accordance with the seven-abstraction-layer Open Systems Interconnection (OSI) conceptual model developed by the International Organization for Standardization (ISO) and the predecessor of the International Telecommunications Union (ITU), published in 1984, and is managed by the ISO/IEC 7498-1 standard.
In addition, Anthony reports that Frakes likes virtualized computing because they and their users can get a snapshot of their systems and software-based virtual machine before applying software patches, which makes it easier to test and monitor the patches before applying them. "We can also make virtual clients run on virtual machines, so if a client crashes, we can bring in a version of the virtual machine on a USB stick, with the last version of the right software, and get it back up and running in a few minutes," says Anthony. "All of this happens on the Ethernet backbone."
Ironically, Ethernet even makes it easier to examine and test its own networks. For example, Jensen International Inc. is a diversified manufacturer with two campuses in Coffeyville, Kan., including a metal fabrication facility. It operates a mix of Ethernet, wireless and wide area network (WAN) connections, but has limited IT staff to ensure connectivity. In the past, Jensen's network administrator, John Heidelberg, carried a laptop to run individual network tests and solve issues, but it was a time-consuming process.
"It's a real pain when you have to check a connection, map the link back to the router, unplug it, go back and then test again," explained Heidelberg. "With some of the networking between buildings carried out wirelessly, troubleshooting can be really tedious. And, when someone calls and complains they can't connect to the server, you have your work cut out for you. Where are they? Where's the server? What segment is it on? Is this a link issue or a protocol problem? Are they getting a DHCP address? Is DNS resolving? It just takes a lot of time to work through."
To accelerate these tasks, Heidelberg recently started using a LinkRunner AT network auto-tester from Fluke Networks, which can perform six Ethernet tests in less than 10 seconds (Figure 4). Its one-button AutoTest feature immediately provides data about a connection's continuity, link/speed/duplex, name and IP addresses (including IPv6) of the nearest switches, DHCP and DNS server availability and performance, Power over Ethernet (PoE) performance, and resource connectivity via a TCP port open or ping.
"LinkRunner AT is fast and thorough with the Ethernet connectivity tests I need, so it speeds up troubleshooting and gives me confidence in the results" says Heidelberg. "I can even get someone who isn't an expert to run a test with LinkRunner AT when I'm not in the office, which would have been impossible before."
Melding for virtual, cloud and IIoT
Since Ethernet already underpins so much of today's digital networking, Internet and wireless, it's no stretch to say it will also serve as the foundation for IoT and IIoT. "Ethernet is the key driver of IIoT, and is at the convergence of operations technology (OT) and IT, too," says Ralf Neubert, senior director for innovation and technology in Schneider Electric's industrial business unit.
"A key theme of IoT and IIoT is network technology and topologies, for example, pervasive Ethernet through to the plant floor and in its equipment," explains Paul McLaughlin, chief engineer at Honeywell Process Solutions. "The latest devices and skids are becoming Ethernet connected with a variety of industry-standard protocols, and so what used to be managed through hardwired I/O signals is being replaced with digital messages and commands. Automation systems are becoming intrinsically intelligent by using OPC UA to join equipment to automation controls and applications. Our LEAP paradigm shift is an example of this effort to simplify device integration."
In this latest installment of Control's Essentials series, the editors tackle the basics of Industrial Ethernet--what it is that makes the world's leading networking technology increasingly suitable for use in plant-floor applications. We review the many protocol "flavors" of Industrial Ethernet that have evolved over the years to overcome limitations of standard Ethernet. Also discussed is the need for industrial hardening of network components such as switches, cordsets and remote I/O modules.
Amazingly, one of the byproducts of expanding Ethernet networks is vastly expanded connectivity, and further consolidation of former network hardware. This goes way beyond early savings in wires, and shrinks the presence of even the latest Ethernet devices added most recently. Hamilton explains that when Grantek worked on an infrastructure refresh of a large soft drink manufacturer's syrup facilities during 2010-15, they first migrated from old Nortel switches to standard Cisco 3860 switches, which allowed them to migrate all control functions from 10 HP manufacturing servers in 20U of rack space to just three HP blade servers in only 9U of rack space. This model also allowed for shared hardware between the business services such as files servers and the manufacturing servers, increasing the overall cost-savings and supportability for the business.
"These functions included primary and secondary SCADA, GE Historian, MS SQL server with SSRS reporting, Proficy Change Management, and two or three terminal servers for GE iFix clients and any custom software required by the site," says Hamilton. "However, what's more incredible is we're now in the process of starting the next refresh, and putting all three servers into a Nutanix hyper-converged chassis that takes up only 2U of rack space because the formerly separate data storage module is now rolled into virtual, software-defined storage. That's a physical consolidation of 90% in just over five years."
In addition, as more drives, switches and other device-level components gain Ethernet ports and access, Anthony reports the ways they show up on networks and even how they're controlled begins to resemble other IT-based devices. "We were doing a production cooling tower recently for an aluminum casting and extruding company, and all its formerly hardwired devices now had Ethernet," he says. "The tower required three 200-hp drives and four 75-hp drives, but the Powerflex 400 drives we installed from Rockwell Automation had Ethernet, so they could be configured and displayed just like any other I/O point on-screen. So, where we used to need more hardware to get more information than just on and off, Ethernet can give us all the data that's available from the drive, such as what faults are happening where, amps and volts being pulled, and everything else about that drive."
Figure 1: Fortescue Metals Group Ltd. uses Profinet to link 15,000 I/O points controlled by 200 Profinet devices, 40 PLCs, 50 HMI workstations and 15 servers, which manage two iron ore mines, three primary and secondary crushers, two ore processors, 15 km of overland conveyors and a two-canyon stockyard at FMG's Solomon Hub iron ore mine in Australia's Pilbara region. Credit: FMG
