Check out the 12 days of cybersecurity mini-series!
Jim LaBonty speaks like the experienced, calm, matter-of-fact process engineering veteran he is, so it's unexpected to hear an edge of concern in his voice.
"The NotPetya malware and attack in 2017 took out Merck's production for two months and cost $870 million. In a brief to Pfizer's board, we estimated a similar incident would cost us $1.5 billion," says LaBonty, director of global technology and engineering at Pfizer, who spoke at ARC Industry Forum 2019 back in February in Orlando. "The financial impact is obvious, but there's a lot more at stake here, too.
"Pfizer delivers about 74 billion doses of medicine per year, including more than 67 billion solid oral doses, 2 billion sterile injectables and 135 million vaccines. It might seem like hospitals and pharmacies would have months of medications stored up, but many providers only have about a two-week supply of many medicines, so an interruption in production could be catastrophic for much of the U.S. health care system."
This is why Pfizer needed to better secure its supply chain, starting with production and onward to providers and people. However, as usual, there were some initial obstacles to overcome. First off, each of Pfizer's 60 manufacturing sites worldwide has a unique combination of:
- Size from small to extra large;
- Pharmaceutical types including biopharmaceutical, chemical, vaccine, solids and sterile injectable;
- IT and automation technology (AT) engineering team capability;
- OT technologies from different control supplier and OEMs;
- Manufacturing local area network (MLAN) and control local area network (CLAN) that range from flat to co-mingled to separated; and
- Push or pull styles of technology integration.
"We're not invincible," explains LaBonty. "We had very poor visibility of our industrial control system (ICS) assets; 7,000 ICS information technology (IT) devices, including many that couldn't be patched anymore and many at end of life; 24/7 operations at 60 plants, making it difficult to schedule downtime for patching; and limited IT and operations technology (OT) governance that increases recovery times."
Evaluate and organize
To continue its mission as a trusted pharmaceutical provider, Pfizer launched its ICS Cybersecurity program in February 2018 to achieve overall control system hardening and rapid threat detection at its 60 plants. LaBonty reports the key objectives of Pfizer's program are:
- Mitigate unpatchable IT and OT assets and related risks;
- Gain rapid visibility of ICS threats and the ability to detect IT and OT malware;
- Update complete inventories daily of system assets and status of their vendors, models, operating systems, versions and firmware;
- Protect manufacturing platforms from intrusion and attacks by known and unknown malware;
- Limit paths and traffic to reach critical manufacturing systems via networks, USB or other means; and
- Make certain ICSs in the OT layer have no direct connections to or from the Internet.
"To make our assets and systems as secure as we can, we need close-in looks at all of them and their control systems layers at our production plants," says LaBonty. "We also need daily updates about what assets we actually have because maybe next week a new suite is added, and the old one is swapped out. To be really protective, we can't work with old data. We could follow IT and patch like crazy to protect against attacks, but we asked if there was a better way, and I think we've got a solution. Still, we need to limit avenues to the factory floor, and limit data that gets down to the ICS to just what's need to run and optimize the plant."
Beyond adopting more heads-up cybersecurity software, LaBonty reports Pfizer's ICS Cybersecurity program also aims to improve on the ISA99 cybersecurity standard's organizational layers with added network segmentation where needed, but also by combining OT and IT tools where useful.
"We learned that we needed to migrate what IT and OT each do well, but also segment them to narrow the network paths and attack vectors to the few critical platforms needed to operate our systems. We use 10 to 20 firewall rules, not the hundreds or thousands that IT uses. We're just saying 'this server can talk to that device,' and that's it."
The Pfizer ICS Cybersecurity program includes seven workstreams:
- Global manufacturing security policy,
- Manufacturing security organization,
- Security awareness/training,
- Risk management
- Preventive and detective controls,
- Asset discovery and threat detection, and
- Vulnerability management.
Besides installing added firewalls, the program also initiated pilot programs for passive network threat detection and asset discovery, OT hardening and whitelisting ICS OT assets, and USB device lockdowns and security, which are being carried forward as full-scale programs that will be built and optimized during the next couple of years.
"We're working quickly and making good progress, but people are the base. That's why our first four workstreams are organizational, and consist of training, building skills, developing a security champion matrix across all sites, connecting to our security operations center (SOC), and enabling policies and structures for cybersecurity governance," says LaBonty. "The last three workstreams are technical, where we're focused on automating asset discovery and threat detection. I'm from the OT world, so I'd rather have a tool that identifies assets, so I don't have to walk around everywhere, and can focus optimizing processes, instead of maintenance tasks like gathering, logging and keeping track of data.
"We'd also rather use tools for asset discovery, threat detection and vulnerability management, so we kicked the tires on them, and found out what worked well and what didn't. It helps that the suppliers that offer these tools are very good, so my best advice is to just pick one, and work with them to secure your asset base. Waiting for nirvana and the perfect solution is waiting too long."
Act and implement
Once the ICS Cybersecurity program's structure was in place, LaBonty reports that Pfizer began segmenting its IT and OT networks, and separating its formerly joined MLAN and CLAN at its plants with firewalls and restricting the avenues between them (Figure 1).
Figure 1: One of the primary strategies Pfizer's ICS Cybersecurity program is using at its 60 different pharmaceutical plants is separating its formerly joined manufacturing local area network (MLAN) from its control local area network (CLAN) with firewalls, and limiting the networking paths between them, so that only authorized communications and essential data needed to run and optimize the plant are allowed through.
"We knew that we absolutely had to segment our IT and OT networks, so we went to the sites with the biggest supply chain or business risks, and tackled and completed their segmentation in about six months during 2018," says LaBonty. "Since we kicked off in 2Q18, our cybersecurity team has grown to about 60 people, and includes three Pfizer people, but mostly consultants and vendors. Is this a highwire act? Yes, but we learned to play well, not get burned, trust that our suppliers will deliver, and improve our cybersecurity with minimal risk."
LaBonty adds a crucial aspect of Pfizer's threat-detection workstream in its ICS Cybersecurity program is passive detection that can view its operations and security measures in real-time. "If you and your cybersecurity tools can see production and network traffic while they're happening, then you can be smarter, and know, respond and plan better. You an even let intruders and malware look around a bit, so you can see what they're after before they do anything bad."
Meanwhile, the program's second technical workstream, OT hardening and whitelisting ICS OT assets, relies on the fact that normal operations are relative stable and don't change much. "Usually, what a controller talks to its devices about doesn't change, so if something starts talking against the baseline, then that's something to look at. Again, the key is making visible what's been invisible in OT. If you're going to invest in segmenting a network, then invest in one more tool that will let you see it. It has always been able to view it's network activity, but OT didn't have to as much because it didn't used to get attacked. Well, now we're front and center on the cybersecurity radar screen."
Finally, USB device lockdowns are the third technical workstream in Pfizer's ICS cybersecurity program because 90% of all intrusions still come from the IT side to the controls environment, according to LaBonty. "We still use some USBs with an added security layer, and this is because we allow zero communications and data to go though cellular modems," he adds.
For example, on a representative aseptic production line, LaBonty showed that Pfizer can view ICS and IT assets discovered at the manufacturing executive system (MES), supervisory control and data acquisition (SCADA) and programmable logic controller (PLC), and equipment levels, as well as communications and data flows. "Plus, we can filter our displays, so they only show OT activity," says LaBonty. "Previously, we'd have to find logical flow maps. Now, we can see what's talking to what, and when a new PLC is added or swapped, it's detected and displayed in two seconds, including a complete audit trail. If a device is added or a wire us changed, you know it.
"This is all a little bit Big Brother, but it's good because we can operate our plants well. Many users change their behavior and culture based on the possibility of being watched. If someone may see them, people will think before doing something, and that's good, too. Viewing our OT assets also gives us insights, instead of having to scramble to check and confirm what's happening. The tools also recommend what we should work on, and takes out 80% of our former effort, so we can spend that time optimizing our processes and saving money."
Communicate, collaborate, trust
LaBonty adds that Pfizer selected Rockwell Automation as its global network services partner because of its global reach, experience, and local country support near many of its facilities, including a central project management office (PMO) with three regional project managers in close contact with its plants. Services provided by Rockwell Automation included design, global implementation, global logistics, PMO coordination, documentation, testing, local site training, and ongoing OT network services support.
"This gives us timely, effective communications, as well as closer partnerships and trust," says LaBonty. "Segmenting our IT and OT networks, adding cybersecurity infrastructure, and migrating assets down to the control system network layer clears up the mud on former IT vs. OT issues, and clarifies roles, scope and other broad areas, so we can make headway quickly. We're establishing separate global IT active directory (AD) domains and local manufacturing site OT layer AD domains, and adding tools to manage them that also setup account and audit trails."
"All of these asset discovery, threat detection and vulnerability management tools are important because many of Merck's assets were wiped out in two hours. If you don't have some cybersecurity protections in place, by the time you've figured out you have a problem, it will be too late, and you'll be toast."
In addition, Pfizer has been working with four-year-old Digital Immunity, a bioinformatics-based cybersecurity provider, which develops in-memory, runtime security software. Its DI Protect product runs in the kernel to harden operating systems and related applications to prevent malware attacks.
"We identify trusted operating systems and related applications on the Microsoft platform, and harden them," says John Murgo, CEO of Digital Immunity. "No other solution provides in-memory, runtime protection using bioinformatic DNA mapping. This means we can stop attacks without any pre-knowledge of the attack on operating systems and applications by building a shield around every process and binary. So far, we've detected 100% of exploits against vulnerabilities."
Murgo adds Digital Immunity can help resolve the traditional conflict when IT tries to apply software patches and OT refusing due to downtime and safety concerns. "We can help defer patching and protect un-patchable systems because our software is more active and uses a more deterministic approach to identify which ones and zeros are good or bad," explains Murgo. "For instance, many OT and IT applications and supply chains can't have downtime and can't deploy antivirus software on the OT side because they can require shutting down production lines during operation. DI Protect is more lightweight and surgical because it runs in the kernel-level at Ring 0 of Windows, while most other software runs at Ring 3. This is really the only way to address this problem because it protects the memory where the code is executed by preventing executable payloads. For example, it would have prevented the Triton/Trisis code from executing."
To protect process applications and their endpoints, DI Protect functions by verifying code loaded in memory, comparing it to each process "DNA map", and "walking the stack backwards" to check any executable code that's trying to run. "We only add a few microseconds of overhead, but this lets DI Protect see any bad code that's been injected, and kick it out," says Murgo. "This lets good processes and applications keep running by eliminating bad processes."
