Last week, the utility met with one of their major ICS vendors to determine if the vendor would be willing to support the utility's test bed concept. The purpose of the test bed is to maintain or improve reliability with security being a potential impact on reliability not the traditional security for the sake of security paradigm. The attendees at the meeting were the utility's Operational Technology (OT) manager, a utility engineering supervisor, the ICS vendor's security manager (not an ICS expert), and myself. Because what occurred is common to many ICS vendors, I will not mention the ICS vendor's name.
The meeting started with the ICS vendor's security manager providing us the ICS vendor's philosophical approach to security and the ICS vendor's overall plan for securing their products. Reliability was not mentioned.
The first question the ICS vendor asked was if the utility has kept current with the latest security patches. (This is a much more difficult question to answer than it appears. Patching is not only temporal -how do you know if you missed the latest, but patches may not be identified as security.) The answer was the engineering supervisor did not know. The engineering supervisor mentioned they have trust in their ICS vendor and want to maintain the equipment warrantee and consequently have implemented all ICS vendor recommendations. The ICS vendor‘s technical documentation did not identify any recommendations specifically as security patches. Consequently, even though the utility was up-to-date with all patching including security, the engineering supervisor did not realize that security was included. This ICS vendor's approach is different than some other ICS vendors that send security patches directly to the end-user's IT security organization. Because of the cultural separation between IT and Operations, many times engineering does not know security patches have been implemented by their own organization.
Some of the security patches actually impacted the reliability of the ICS and the overall system. Since the engineering supervisor did not realize they were security patches, he contacted the ICS vendor's engineering team about the impact to the reliability of the systems. The ICS vendor's engineering team also did not realize they were security patches and consequently the message was not relayed to the ICS vendor's security team. Thus, the fact that a security patch could, and did, harm the operation of the system came as a surprise to the ICS vendor's security manager.
As previously mentioned, when the ICS vendor discussed their "long-range" approach to security, it did not reflect reliability considerations. Consequently, when the utility explained the purpose of the test bed, the ICS vendor asked if the utility would be willing to partner with the ICS vendor on improving the security/reliability of their products. Tentatively, the utility has agreed pending formal legal approval.
There are a number of very important generic aspects to this discussion:
- Security and reliability need to be addressed in a comprehensive technical manner
- Security and reliability can be mutually exclusive potentially making a "secure" ICS less reliable
- For ICSs, all patches should be for reliability with security just being one aspect
- For ICSs, patches that do not affect reliability or safety should NOT be implemented
- There often is a disconnect between the utility's Operations and Security organizations
- There often is a disconnect between the ICS vendor's Operations and Security organizations
- Security patches are not always identified by ICS vendors as such leading to unintended consequences that are not easily identified in a root cause analysis
- Security patches that are implemented by the end-user's IT organization without input from engineering can also cause problems that are not easily identified in a root cause analysis
- There are very few organizations willing to look closely enough to even identify the issues identified here
There will be a session on the status of this effort at the October ICS Cyber Security Conference (www.icscybersecurityconference.com)
Joe Weiss
