OT control system field device cybersecurity issues are different from those affecting Internet Protocol (IP) networks. These differences must be understood by any organization creating OT/ICS cybersecurity policies or recommendations. Too often, government and industry policies for OT cybersecurity focus on IP networks and assume that the policies apply to all OT, including legacy ICS field devices and control systems. From a cybersecurity perspective, legacy control systems are not just legacy pneumatics and 4-20 milliamp analog sensors, but also “modern” digital devices. In essence, all field devices currently deployed for control systems are legacy systems with little to no cyber security. Furthermore, the lack of senior management involvement of engineers in the development of cybersecurity policies, as documented in my article in the May/June 2020 issue of PE Magazine “Attention to Policy Makers: Cybersecurity Is More Than an IT Issue”, unabated.
The US Government Accountability Office (GAO) report GAO-19-332 states:
“To exacerbate the risk associated with the increased attack surface, many legacy industrial control systems were not designed with cybersecurity protections because they were not intended to connect to networks such as the internet. For example, many older devices are unable to authenticate commands to ensure they were sent by a valid user and may not be able to run modern encryption protocols. Additionally, some older devices are unable to log commands sent to the devices, making it more difficult to detect malicious activity. In addition, even with more advanced devices, the safety and efficiency goals of the network and supporting industrial control systems can conflict with the goal of safety in the design and operation of industrial control systems. According to an analysis by the Idaho National Laboratory, network owners and operators may not always be able to identify vulnerabilities in industrial control systems in a timely manner. Vulnerability scans are commonly used in IT systems to validate proper system configuration and identify any vulnerabilities that may exist. However, traditional IT vulnerability scans can disable or shut down power systems, and testing may not always detect vulnerabilities deep in the software of industrial control systems. Even if owners and operators are able to identify industrial control system cybersecurity vulnerabilities, they may not be able to fix these vulnerabilities in a timely manner because certain industrial control system devices may have high availability requirements to support network operations. These devices typically need to be taken offline in order to apply patches to address cybersecurity vulnerabilities. In addition, network owners and operators must thoroughly test the patches before applying them. Security patches are typically tested by vendors, but they can affect or change the functionality of industrial control systems, which can have serious consequences for network operations.”
As noted in the GAO report, there have been many documented instances where the application of IP network mitigation measures has resulted in significant problems with control systems and control system field devices.
- IT patches have compromised control systems and even created security issues. This included a patch for a turbine control system that was not coordinated by OT (square pin) and Engineering (round hole), although the patch was tested by the network organization before being sent to the customer. The untested system interaction resulted in the turbine control workstation no longer being visible and the turbine having to be shut down. However, the unintended system interactions of the “untested” patch prevented the technician from shutting down the turbine from the technician’s workstation – a major security concern.
- IT penetration tests in control system networks have resulted in shutdowns or damage to control systems and control system communication. In one case, a utility company’s IT security group (square pin) scanned data center facilities with IP network scanning software, and then extended the scanning to large substations (round hole). The security group had no prior experience scanning substations. After the scans, the relays showed problems, but SCADA was unaware of the problems. The port scanning of this new tool resulted in the real-time protocol operation of the relays being halted and operation on the CPU being suspended (two different relay vendors), leaving the DNP/non-real-time operations alone – the worst possible circumstance. To clear the error, each relay had to be shut down and restarted to restore operation. Several hundred relays were affected. All devices in each substation were affected at the same time. Unaware that a security scan was initiated, it looked like a distributed denial-of-service (DDOS) attack, causing the devices to malfunction. A grid failure with unavailable high voltage relays could have resulted in a large nationwide outage that would have damaged many large transformers and customer equipment. In another case, IT (square pin) conducted a penetration test and caused a denial of service to 6,000 control system devices (round hole). It took a total of 15 days to reset each control system device.
- Network mapping tools can affect control system field devices. In another case, frequency converters were connected to the network. The network mapping tools (square pin) caused a buffer overflow that resulted in a hard drive failure requiring a shutdown and hardware damage requiring replacement of the configuration modules (round hole).
- The application of anti-virus software (square pin) to many legacy distributed control system DCSs (round hole) has caused denial of service conditions.
- System hardening applies to Microsoft Windows-based devices (square pin). However, most field devices for legacy control systems (round hole) do not use Windows.
These and many other cases demonstrate that IT security technology (square) that works well in a constrained IT environment may not work well in a constrained OT/control system environment (round hole), especially with legacy control system devices and communication protocols. There are network security tools specifically designed for use in control system environments that work well. But even these must be tested before use with older legacy control systems.
I remain very concerned that both private and public sector organizations that are policy makers (square) simply do not have the technical depth of control systems cybersecurity to make decisions about control systems cybersecurity (round hole ). This is not just a US problem. Recently, for example, the German Cybersecurity Policy Setting Organization (square pencil) conducted tabletop exercises focused on power generation without input from the power generation engineering organizations (round hole). Control systems cybersecurity training, covering unique topics such as process sensors, system interactions and common cause failures, is required to educate both the workforce and policy makers.
These issues between square peg networking and round hole engineering will be the topic of my presentation on October 26th in Minneapolis (https://www.cybersecuritysummit.org/).