Greg: Plant performance not only depends upon the best measurement and control system design, installation, commissioning, maintenance and operability, but also upon the acceptance and commitment of plant staff. Many applications are challenging. Mammalian bioreactors have exceptionally stringent measurement and control requirements, particularly with pH. Deviations as small as a tenth of a pH from best values can cause a significant decrease in cell growth rate or product formation rate. Here and in all process control applications, better performance depends upon the ability of practitioners in the field learning and addressing the many issues involved. Often overlooked is the need for good communication with the plant staff so that they have confidence and involvement in getting the most out of the automation system. It is critical to be open minded and seek the best solutions and fixes based on what you see in the plant and what you hear from the plant staff. Also, companies may not understand or be looking for process control improvements. We are fortunate to have David Bruton, automation technical manager at Veqtor Solutions, share his personal experiences and technical achievements in mammalian bioreactor control that include making the most out of personal relationships with the plant staff.
What roadblocks do you come across with testing process conditions in bioreactors?
David: Bioreactors at pharmaceutical companies have very strict requirements when it comes to making changes during a Good Manufacturing Practice (GMP) batch. And for good reason! These products are often injected into patients, many who are immunocompromised.
I always start by talking with the manufacturing teams. They’re the ones that know the process forwards and backwards. They live with the process every day. Typically, when you ask about putting control in manual for tuning, you’ll get a warning that any manual intervention is considered a deviation – regardless of how much the process values deviate. You may get referred to the Quality Assurance department, who will reinforce that those interventions during a GMP run will require hours if not days of extra time to assess and document the impact to the product. So, it’s best to talk with our friends in QA before making any changes to avoid the extra headaches and investigations.
Greg: How did you deal with the need for best tuning within these restrictions?
David: Because so little leeway is allowed during GMP batches, it’s important to take advantage of every opportunity outside of these runs. If you know of an engineering run or water run performing other tests, use this opportunity to schedule some time for tuning as well. Engineering runs with cell culture are especially valuable because you can measure a more accurate pH response.
Greg: What about when a plant is running with no test runs in sight?
David: When plants are up and running, there may be no opportunities for tuning in the foreseeable future. This is when the value of the digital twin simulation can truly shine. Testing out the control response before running it on a multi-million-dollar batch might give you, and your boss, that reassurance you need.
If you don’t have a digital twin with your expected process response, then it’s especially important to have friends on the manufacturing and QA teams. If the process control is failing, a deviation may have already been opened. You might make the most of this situation by tuning the process as a temporary or root cause for the deviation. And if you’ve been looking forward to fixing this problem, you might just be happy to get that after hours call.
Greg: pH control in bioreactors often use sodium bicarbonate. Mammalian cell cultures are sensitive to sodium concentration buildup as they lead to an increase in cell osmotic pressure and membrane rupture. This makes split-range crossings especially important to manage in pH control. How did you minimize the common problem of unnecessary crossings of the split-range point?
David: Absolutely, it’s quite important to minimize the crossing of the split range in pH control. Thankfully there are several tools to address it: a split-range gap, disabling integral action in that gap, an overdamped tuning response and directional velocity limits.
Greg: Well, it certainly sounds like you made your way through the Checklist for Bioreactor Control. When do you use a gap in the split range? And how do you disable the integral action in it?
David: Yes, that checklist is a great place to start. A split range gap is used whenever the use of the two reagents is to be minimized. For example, if 50% is the appropriate split, then the PID’s output from 0-48% would be to add base and from 52-100% to add acid. There is a gap in the middle of the output range. This allows some additional time before the crossing from acid to base occurs—but it only delays the crossing.
To prevent this, the integral action can be disabled when the PID output is near the split range gap. The common PID feature “integral dead band” can be conditionally enabled when the PID output is near the gap.
Greg: How about the tuning response? How did you guarantee an overdamped response?
Oscillations eliminated and tight control achieved by tuning
David: As seen online in Figure 1, using integrating tuning rules with a first-order-plus-dead-time model, it is possible for a load response to return to setpoint with no overshoot. This is especially the case when pH is cascaded to carbon dioxide. By choosing an arrest time much greater than the loop’s deadtime, the process will gradually return to setpoint without overshoot. Usually, batch run times are unaffected by the few extra minutes required for the tuning response because mammalian bioreactor batches run for days.
Greg: You also mentioned some directional velocity limits. How can these help in pH control?
David: Directional velocity limits are an additional level of control used to slow down the PID response in a chosen direction. It can be used to slow down how quickly the base dosing is increased, while allowing the base dosing to decrease quickly. If you’re using velocity limits, it’s important to enable external-reset feedback (e.g., dynamic reset limit) to prevent windup. Now, if you’re investigating your base addition and find that the sodium bicarbonate system doses the same amount regardless of the PID response, you have found a major problem.
Greg: Sounds like you’ve dealt with that issue before. How did you resolve that?
David: I have indeed. One way to deal with this issue is to add a pulse-width modulation. When there’s only a discrete valve, then cycling the valve’s open time based on the PID response will help to linearize the addition. Be careful as this can lead to additional wear on the equipment.
Greg: What are some of the other improvements for the pH control?
David: PID controllers expect a constant process gain. If the acid and base doses have significantly different process gains, try implementing a gain scheduler or adjusting the split-range point. For the same reason, it is beneficial to use weak acids and bases and low concentrations of sodium bicarbonate. This is especially important for smaller reactors, where stronger reagents and more concentrated doses have a greater impact.
Instead of controlling to setpoint, many sites shut off the pH control when the pH gets within a dead band of the setpoint. While this is effective at reducing sodium bicarbonate additions, it ignores the integrating nature of the pH process and the interactions with the dissolved oxygen control. Integrating processes, by definition, increase or decrease when there is a mismatch between the process input and output. The interaction from the dissolved-oxygen control can displace necessary carbon dioxide. To maintain the setpoint, some carbon dioxide needs to be added to make up the amount getting displaced.
Greg: That’s quite a big problem from a misunderstanding of the process. What did you do to maximize communication and acceptance by plant staff?
David: I love to help my customers, and I enjoy teaching them what I’m passionate about: process control. I’ve led trainings from introductory process control and dynamics to advanced tuning. I also find it best to work as part of the plant team rather than as an outsider. I typically walk the team through my process and discuss the pros and cons of different approaches and designs. You’ll earn a lot of trust that way. Sometimes the best approach is listening to the plant staff to make sure their intended design is heard and understood. Even then, you may face strong expectations for things to stay as they have been. It can be challenging to break through without enough time and understanding on both sides.
Greg: What can we do to increase a company’s interest in process control improvements and addressing the challenges for accomplishing results?
David: It’s important to emphasize the progress made, and make it known to everyone impacted. I find it’s quite helpful to have before-and-after snapshots that clearly illustrate the impact of the change. Show them to a few engineers who spend all day looking at problems and celebrate some good news together. Don’t forget to tell the manufacturing teams who use it. That way, your changes will be put in service and the gains will be realized. When you share it with the engineering managers, try wording the improvements it in terms of cost savings. That will make it easier for them to pass it along to their higher ups and justify the extra expenses that your raise deserves.
Greg: For more on biochemical reactor measurement and control and how the digital twin can play a key role, see my ISA book New Directions in Bioprocess Modeling and Control – Second Edition.
David: And if you’d like a second set of eyes on your control problems, whether tuning or batch related, I’m always looking for new challenges.
10. Get to work in interesting places like Ireland
9. Learn patience since bioreactor batches can take weeks
8. Time to get top know operators while waiting for batch to complete
7. Realize value of future value block to analyze batch profile and predict batch end points
6. Gain experience with use of external-reset feedback to stop oscillations
5. Discover how simulation with speedup can help detail and test improvements in less than an hour
4. Get great at tuning controllers
3. Achieve incredibly tight pH and temperature control with no overshoot
2. See benefits in millions of dollars
1. Become a hero
