Solutions spotlight: Phillips 66 modernizes DCS in face of pandemic

Keith: Yeah, I understand Bayway was already in the midst of a multi-year migration of your legacy Honeywell DCSs to the latest Experion virtualized environment when the pandemic hit. That sort of all-encompassing migration is a challenge in the best of times, much less in the midst of travel and social distancing restrictions imposed by the pandemic. Can you give our listeners a sense of the scope of the migration project and how you've managed to really forge ahead in this time of adversity?

Patrick: Yes, of course, Keith. So Phillips 66, the refinery up here in Linden, New Jersey, it is a multi-year control system migration, I like to call journey. Our facility is operating 365 days a year, 7 days a week, continuously running, and we're processing about 300,000 barrels a day of crude, making products like gasoline, ultra-low sulfur diesel, jet fuel, a plastics plant on site as well and this distributed control system is the backbone, the brains of our facility. And not only do we have to migrate it on process because of trying to limit downtimes, like you said, these COVID conditions kind of put a hamper in it, right? So, our system is a legacy coaxial cable LCN-UCN system. We're gonna be going to the latest  Experion virtualized environment.

So this coaxial cable is not going to be supported after 2023, 2024. And, with the new control system we're leveraging not just the virtual environment. And then we'll talk about the remote factory acceptance test with there, but we're actually leveraging big beefy servers, virtualizing our equipment, virtualizing nodes, minimizing, or trying to think forward and really minimize the maintenance going forward for our system.

So, our system has been running and really reliable for the past 20-30 years and converting it to something that could last another 20 or 30 years, that's the ultimate goal, right? You wan to get there. So, when the pandemic struck, social distancing and travel restrictions, we still needed to progress the project, right? We have a tight timeline to work against and we leveraged, at first, the open virtual engineering platform, open VP from Honeywell where we've put our system up in the cloud and utilized that system to develop graphics and have an interface that you can test and look at graphics, which is your human-machine interfaces that the operations interacts with online.

So, we were doing that pre-pandemic. But we utilized that throughout the pandemic. And then we went one step further and did a remote factory acceptance test where we had a Honeywell employee in the factory and then we were remote working with them, them having helmet cams, screen-sharing and doing the tests remotely.

Keith: Yeah. I can imagine you did a lot of the software work even before that, before you had the hardware in place in a more virtualized environment.

Patrick: Yeah. So the software, like we were saying, using the open virtual environments up in the cloud, doing all that testing, all those checks really lends itself to doing it remotely. So, a lot of that stuff we were doing pre-pandemic, it kind of just did it tenfold once pandemic hit. So to me, this pandemic, 2020 unfortunate times, but we want to make sure, Honeywell and Phillips 66, and everyone wants to keep their employees not just safe, but healthy. You're using technology now, leveraging technology to continue to progress on this project.

So we kind of tenfold, I said, why not? Let's, you know, we've been doing these things remotely. Why not go further? Why not try new things? And what do we have to lose at this point? And we're leveraging new things that we can continue going forward, cost savings on both sides, both on the customer, Phillips 66 and the vendor itself too, Honeywell.

Keith: Yeah. Can you talk a little bit more about that factory acceptance test? How did that really compare the process to a traditional one, what we're all used to, where you get both the teams together and some hardware somewhere and put all the software and stuff and kiss your family goodbye for six weeks and go off and do the problem.

Patrick: Yes. So, instead of being trapped literally at a factory for those weeks, right, and physically touching the hardware, it is different, but you're doing all the same tasks. So, Honeywell had an engineer at the facility with a helmet cam and gave us live feed of what they're looking at. So, when you go through procedures, whole wires, see all those tests, you get to not only see the screen itself that your system is based off of, but you can actually see what's happening in the factory.

Now, you know, I don't want to say it's not too much different because once you get on-site, we added more steps and checkoffs and checks once the hardware got here on-site. So, it's a little give and take, right? You know you're not gonna be there, physically touch it. You know, what happens if you do this? What happens if you do that? You stick to the procedure, you have what we use in the oil and gas industry repeat back, right? Like we repeat back constant communication and outside of not being there and not physically being able to physically touch it, we were able to accomplish everything we could in the factory.

It's going through all the procedures, the test outs, the checkouts, screen-sharing, verifying. And then we do the final sign-off once the equipment actually gets here on-site. So you're gonna, you know, a bit more time and conserve like a little bit more contingency in the front end of your site acceptance tests. But for a factory acceptance test, you can get 80% to 90% of that material complete flawlessly with remote services.

And I will say, Honeywell really stepped up to the plate for the factory acceptance test. They had cameras on the floor, cameras attached to them, to their helmets on the actual factory. The engineers at the site were fantastic in the factory. And one thing that you can leverage now with remote work, you can have experts all around the globe, right? So, for folks doing advanced solution packages, specialty things, any type of custom specifically for a site, you can pull those experts that won't necessarily be there in the factory, but be there while you're going through the procedure and while you're going through all the checks and balances at the factory.

So, it is a little bit different. I will say I won't say you can 100% get there, but you can do a lot of it. And I think especially on the software side, I can comfortably say if all, if not most, can get there and hardware, yeah, you really just got to be smart, have checks and balances when the hardware gets on-site to make sure that everything you did test is actually physically there. And the vendor did a great job allowing us to actually feel like we were there, but virtually.

Keith: Any particular challenges you could warn our other listeners about before undertaking, things to watch out for that maybe you didn't expect challenge-wise?

Patrick: I think the vendor and us were exceptionally prepared coming in because we knew we'd prepared ourselves. We knew this was going to be new. Something that the industry hasn't done yet, at least for these types of on-process migrations, and testing, and new hardware. So, we had a lot of preparation going in. But I think the number one thing, the biggest challenge, and it goes to any project is communication. And when you're virtually and doing remote, desktop sharing, camera live feed sharing, communication is that much more important.

I have a saying: our business requires 100.0% accuracy. If that 0.1% that's not accurate can really cause something really bad to happen. And when you're testing things out, testing hardware out in a factory, you've got that mindset coming in and it almost gets up to 100.00% when you're doing it virtually because you have to be that much more prepared, that much more communication between the two. And, that's the expectation I set for my team. That's the expectation I set for the vendors that we work with including Honeywell.

Keith: Any surprisingly beneficial aspects of doing this virtually that maybe you didn't expect?

Patrick: I think the biggest surprise was not only, and probably the biggest benefit, was the maybe drawing a parallel to my personal life. You know, with COVID, my folks the way they see their grandchild mostly is through FaceTime or conferencing, right? So, there's a lot of things like, you know, FaceTime, but I will say the folks in...my parents had never used it pre-pandemic. So you've got a lot of people who are hesitant to utilize technology because they're not used to it or it's not something they've done before. They're not...you know, there's a hurdle, it's something different.

And my silver lining, at least with the pandemic and this project was, you know, we're doing things that we haven't done before, we're progressing the project, keeping it on schedule and on budget, and we've got people now that want to leverage this technology and say, "Okay, well, you know, I spent weeks away from my family traveling for this factory acceptance test," you know, instead of saying, "What can we do remotely?" It should be questioned, you know, what can't we do?

And that's kind of the challenge I put on them and myself of, prove to me why we have to travel. And that's the mindset this pandemic had put on us. But I think it's something that is a fair question. And I do really enjoy people adopting technology in the controls world. Us controls folks, I think we've been doing digital transformation before digital transformation was a buzzword. But I do think, it is our time to shine in this pandemic where the silver lining is just the adaptation of technology, the acceptance of it, the acceptance of all these virtual events, the virtual conferences, the virtual factory acceptance tests to kind of put it in overdrive in terms of speed of adoption.

So that's what I'm pleasantly surprised for in the management, like the management acceptance of it and support of it. They want to see this system succeed. They want to see this system progress, the project to progress. And they were super supportive of us doing something a little different, a little bit outside the box and prove to them with the checks and balances that we've had.

Since that remote factory acceptance test, we've had a site acceptance test and we've physically cut over all that hardware that we test in the factory virtually. So, it's all proven out. We've had a success rate all on-process migration, and I'm super proud of the team, both on my team at Phillips 66 and our partners in Honeywell implementing this.

Keith: Sounds like you had just a cut over some of that stuff early this morning, yes?

Patrick: Yes. 3:30 this morning. So, about three and a half hours ago, I came in and were physically on all that experience on all those months of work and years of work and preparation for that one day of cutting over. I think my favorite story is when we did our first LCN cut over to Fault-Tolerant Ethernet, virtual servers and,  the Honeywell team and the Phillips 66 team on the control side had a little golf clap. And all you heard was like clap, clap, clap. Like it was this great comradery moment, where like this month almost blood, tears all come over and it's such a great feeling, but yes, today, we did just cut over the equipment that we did virtually factory acceptance desktop.

Keith: Well, great. Congratulations. Glad to see you're still smiling. Can't have gone too rough. Yes, absolutely.

Patrick: So far, so good.

Keith: So, it sounds like this new methodology is something that was very positive for Phillips 66 and methodology you will likely continue using when you don't have to, let's put it that way.

Patrick: Yeah. We were able to keep our team safe while maintaining schedule like this piece coaxial cable systems aren't manufactured new anymore. The support is going away. So staying on schedule for the cut-over is important, but the most important thing is keeping our folks safe and healthy, especially with everything that's going on in this world. And yeah, it really proved locally, corporately that these things are feasible and where else could you leverage it? Right. I think the controls world leveraged at first, but remote factory acceptance tests, I mean, those could be the wave of the future in terms of receiving equipment before it comes on-site.

Keith: Well, it certainly makes sense that by decoupling the hardware and software, you've got some real opportunities to do that much, you know, that's the wave of the future and the architecture of the future.

Patrick: Yep. Yeah, not just the human-machine interfaces, but the second you start virtualizing your environments and your control systems all those checks and balances become emulated hardware. So, it's software-based and you can go through all those checks and balances. But yeah, you're right, the hardware that you do put on site, you've got to do some double checks when it gets here, but it is well worth it. And it's something we're going to leverage going forward. I don't think we need to, with the open virtual engineering platform, open VP for graphics, that is a seamless process for us, and we're going to be doing that remotely going forward for all graphics and the virtual factory acceptance test, we're going to be leveraging those abilities going forward, too.

Keith: Great. Well, thanks so much for sharing your insights and your experience. It's always very, very meaningful to hear some of these things that we hear about really being experienced and reaping benefits for the end-user community. So, congratulations on that. Thanks for sharing. And for those of you that are listening, thanks for tuning in. Thanks also to Honeywell Process Solutions for sponsoring this episode.

I'm Keith Larson, and you've been listening to a 'Control Amplified' podcast here with Patrick Robinson of Phillips 66, Bayway Refinery in Linden. Thanks for joining us. And if you've enjoyed this episode, you can subscribe at the iTunes Store and at Google Podcasts. Plus, you can find the full archive of past episodes at controlglobal.com. Signing off until next time and thanks again, Patrick, for joining us today.

Patrick: Amazing opportunity. Great work, Keith. Appreciate it. Keep it up.

Keith: Take care. You too.

For more, tune into Control Amplified: The Process Automation Podcast