Standards
Certification
Education & Training
Publishing
Conferences & Exhibits
Improving Water and
Wastewater SCADA Cyber Security
2012 ISA Water & Wastewater and Automatic Controls SymposiumAugust 7-9, 2012 – Orlando, Florida, USA
Speakers: Bill Phillips and Norman Anderson
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 2
Presenter
• Bill Phillips, PE: Bill specializes in delivery of secure
and reliable process control and SCADA network and
communications systems, cyber security vulnerability
assessment, and facility automation and information
system planning and implementation. Bill has over 30
years of process control and SCADA system experience
and has focused on control system network and
communications cyber security for the last decade. Bill
has a BSEE from Clemson University.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 3
Presenter
• Norman Anderson, PE: Norman has over 5 years
experience in the design and commissioning of Process
Control Systems for the Water Sector. Norman has
provided secure and reliable PLC, SCADA, and Network
hardware and software architecture designs and
provided control system automation solutions for a range
of facilities. Norman has an M.S. in EE from Iowa State
University and an M.S. in Physics from the University of
Florida.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 4
Presentation Outline
• Need to secure control systems– Continuing increase in Cyber Attacks
• Notable Cyber Attacks
• Available Guidance and Resources– Standards
– Design Guides
• Assessment/Design/Implementation/Operation– Determining Risk factors and mitigation techniques
• Our Experience and Examples
• Summary
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 5
General Increase In Cyber Attacks
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
Nu
mb
er
of
Vu
lnera
bilit
ies
Year
CERT Cataloged Vulnerabilities 1995-2007
Vulnerabilities
0
20000
40000
60000
80000
100000
120000
140000
160000
19
88
19
89
19
90
19
91
19
92
19
93
19
94
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
Incid
en
ts R
ep
ort
ed
Year
CERT Reported Incidents 1988-2003
Incidents
• General Trend of increase in incidents
and vulnerabilities.
• CERT stopped incident monitoring in
2003.
*Source: CERT Statistics
http://www.cert.org/stats/#vul-year
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 6
Reported Incidents by Infrastructure
Sector
Water/Wastewater is #4 on the list and has twice the incident rate of most commercial
facilities.
*Source: Summarized by Infrastructure Sector (RISI, 2010)
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 7
Industrial Security Incident Attack
Points of Entry
Many attacks are through local business networks and via remote access. These are two common
connections to industrial networks to allow for machines having email and internet access to
connect to SCADA networks and to allow remote vendors to connect to SCADA networks for
maintenance.
*Source: Summarized by Points of Entry (RISI, 2010)
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 8
Financial Impacts
Approximately 23% of the industrial security incidents resulted in damages greater than one
million dollars per incident.
*Source: Reported in the U.S. (RISI, 2010)
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 9
Media Coverage
• Pump destroyed at water plant Springfield, ILo Believed to be due to cyberattack (not confirmed by DHS).
o Story covered by news media such as the Washington Post, Fox News, CNN, and MSNBC
o Even though unconfirmed, the utility was in the national spotlight for weeks
• Texas SCADA system hacked and screenshots of HMI releasedo Response to DHS downplay of IL incidento Again carried by major news mediao Used a virtual network connection with the internet with simple password to access network
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 10
More Infamous Attacks
• Maroochy Shire Sewage Treatment Plant in Queensland, Australia.
o Attack resulted in approximately 212,000
gallons of raw sewage to spill out into local
parks, rivers, and a nearby hotel.
o The attack was perpetrated by a disgruntled
insider and former Contractor, Vitek Boden,
that previously installed the radio-controlled
SCADA equipment for the plant.
o During the attack period, Boden used a laptop
computer and stolen radio on at least 46
occasions to issue unauthorized radio
commands to the SCADA System (Abrams
and Weiss, 2008)
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 11
More Infamous Attacks, Continued
• Stuxnet
• High sophisticated WORM to target Siemens PLCs
• Used to destroy centrifuges used for uranium enrichment
• Deployed using USB flash media devices (thumb drives)
o No external connections does not equal safety
• Showed the weaknesses of Industrial Control Systems
• Duqu (Stuxnet Variant)
• Discovered by Symantec and appears to be a variant of Stuxnet
• Not intended to destroy industrial control systems but to steal information from
them
Native Code Code with virus
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 12
Common Vulnerabilities
• Denial of Service (DoS):
– Attempt to make computer network unavailable
– Would slow or shutdown the communications SCADA network
– Mitigation techniques include Firewalls, ACLs, Intrusion Prevention Systems
• SQL Injection
– Attacks SQL databases using vulnerabilities in websites
– Can steal database information or destroy data
– Mitigation techniques include effective patch management, Intrusion Prevention Systems
• DCOM
– Most notable are RPC DCOM and Blaster attacks
– Can take control of computer and install programs, view, delete, etc.
– Mitigation includes use of intrusion detection, packet filtering, and network segmentation, and port blocking
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 13
Example Control System Attack
Animation Explains Control System Attack By Remote Attacker
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 14
Importance of Security
Why Security is Important at a Water or
Wastewater Facility:
• Critical Infrastructure and Public Safety
o Critical resources
o Downtime can affect life safety
• Operational Reliability and Availability
oAttacks can lead to significant downtime
•Financial Impacts
o Loss of revenue for utility and its customers
o Mitigation and legal costs
•Media Attention
o Loss of public confidence
oStaff intimidation
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 15
Available Guidance
• AWWA Roadmap to Secure Control Systems in the Water Sector published in 2008
o Goal is in 10 years to have no loss in critical function due to cyber attack
o Develops a roadmap with goals at the 1, 3, and 10 year marks. Currently in year
4 (mid-term) of program
• ANSI/ISA-99.02.01-2009 Security for Industrial Automation
and Control Systems: Establishing an Industrial Automation and
Control Systems Security Program
o Builds upon global standards ISO/IEC 17799 and ISO/IEC
27001 and addresses the difference needed for industrial
security
o Defines procedures for implementing and assessing secure
industrial control systems
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 16
Available Guidance, Continued
• NIST SP 800-82
o Final Version Published:
http://csrc.nist.gov/publications/nistpubs/800-82/SP800-
82-final.pdf
o Goal is to provide a guideline for critical infrastructures to
secure their control systems with the idea to maintain
systems online and operating unlike traditional IT systems.
• NERC – Critical Infrastructure Protection (CIP)
o Numbers CIP-002-3 through CIP-009-4 (18 standards)
related to Cyber security implementation plans
o Covers implementation of management controls as well
as operating procedures for personnel
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 17
Available Guidance, Continued
• Cisco/Rockwell Automation – Converged Plantwide Ethernet
(CPwE) Design and Implementation Guide
oProvides design and implementation guidelines
for industrial control systems based on the
manufacturing industry
oGoal is to provide less downtime, higher
security, and optimization of Industrial
Ethernet networks
o Guide provides real network architecture
examples, security methods, and
implementation methods
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 18
Securing Networks
• Securing networks requires proper planning to ensure successful implementation. There are four basic stages of planning and implementation for network security:
1. Assessment• Determine Risks and Mitigation techniques
• Risk impact versus cost of mitigation
2. Design• Develop appropriate network architecture and segmentation
(NOTE : Taylor to selected HMI suite TCP/UDP port requirements)
• Choose necessary hardware and software
3. Implementation• Qualified and certified installers and designers
4. Operation and Maintenance• Develop operational procedures for staff
• Maintain network, hardware, and software
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 19
Assessment
The Critical Starting Point
• First step for proper network security
• Past Assessments were largely based on RAM-W– This method was not very specific or comprehensive
– Limited guidance was available at the time
• US-CERT Cyber Security Evaluation Tool (CSET)– Developed by DHS to assist in protecting key assets with
assistance from NIST
– Available free from the US-CERT website: http://www.us-cert.gov/control_systems/satool.html [training from Control System Security Program (CSSP) also provided]
– Uses 4 major steps and generates a report based on current industry standards
• Assessment is then used to plan and prioritize mitigation solutions
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 20
Typical Large Utility Control System
Network
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 21
Typical Small Utility Control System
Network
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 22
Typical Small Remote Systems
• No matter the size of the network
there are still critical systems to
protect.
• Process control networks are
inherently different than IT
business networks even though
many components are similar.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 23
Wastewater Utility Control System
Design Example
• Includes redundant WAN connections
• Internet connection for WAN extension to remote facilities & mobile remote access
• Compact resilient core network
• Uses VLANs and firewall sub-interfaces to tailor network architecture to SCADA HMI applications suite requirements and to securely support business network access
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 24
Network Segmentation – Using
VLANs
• Network organization secures and helps maintain networks.
• Virtual LANs (VLANs) - Useful for SCADA systems because VLANs define broadcast domains that can be widely separated (i.e. not on the same network segment)
• Can reduce costs, by allowing host on different networks to share layer 2 switches.
• Use 802.1q VLAN encapsulation protocol
• Layer 3 device required to route between VLANs, some Layer 2 devices will support VLANs to some extent.
• VLAN Approach:
o VLAN Range: 1-1005 (normal) & 1006-4094 (extended)
o Don’t Use VLAN 1 (Native VLAN)
o Verify VLAN capabilities of network switches & routers
o Use logical approach
o Incorporate VLAN designations into IP Addresses
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 25
Network Segmentation – Using
VLANs (Example)
• Example:
• VLAN 10 – Network Management
• VLAN 20 – SCADA DMZ
• VLAN 30 – SCADA
• VLAN 40 – Security (Video)
• VLAN 50 – Remote User (DMZ)
• VLAN 100 – Public Media WAN (Inter Facility VPNs)
• VLAN 110 – Backup Public Media WAN
• Extensions: (For shared media)
• VLAN 60 – Business
• VLAN 70 – Business Remote User (DMZ)
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 26
Network IP Addressing
• Approach:• Use 10.0.0.0 private network Class A for primary VLANs
• Use 192.168.0.0 private Class Cs for routed links
• Incorporate facility & VLAN numbers into IP addresses
• Limit broadcast domains to a single facility
• Primary VLAN Example:• 10.VLAN.Facility.Host/X or 10.Facility.VLAN.Host/X
• X = Subnet Mask bit count
• X (between 24 &30) based on anticipated host count
• WAN Example:• 192.168.1.Y/X
• X = Subnet Mask bit count
• (between 24 & 30) based on number of nodes
• Y (between 0 & 252) = Network Number
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 27
Example Firewall Configuration
Specification
• Security Levels - Implicit Deny Lower-to-Higher level
• Interfaces• Typically 3-4 for small to medium size firewalls
• Sub-interfaces can extend that number
• Stateful Inspection• Can drop otherwise legitimate packets that are not part of an active
connection
• Holds in memory variables defining the state of each connection
• State variables include things like source and destination addresses, port numbers, packet sequence numbers
• Access Control Lists• Used to apply access control rules at interfaces
• Format: access-list-number {permit | deny} protocol source {source-mask} destination {destination-mask} [eq destination-port]
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 28
Example Firewall Configuration
Specification
• Security Levels• Each Interface & Sub-interface
• Inside – 100 (Most trusted)
• Outside – 0 (Least trusted)
• DMZ – 50
• Access Control Lists• Permit DMZ –to-Inside SCADA specific traffic such as web server,
terminal server and historian traffic.
• Permit VPN LAN-to-DMZ authenticated remote user traffic such as web server, terminal server and historian traffic.
• Remote PLC Connections:
• Consider a Remote PLC DMZ to avoid direct connections between Internet connected PLCs and the SCADA network
• Consider dual Ethernet DMZ PLC interfaces (i.e. separate VLANs) to increase separation.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 29
Domain Controller Implementation
• Use group policies to manage role based access
• Separate controllers required for each domain
• Domain Controller and Active Directory Traffic• Uses Remote Procedure Calls (RPC) and Distributed Component
Object Model (DCOM) which introduce numerous vulnerabilities.
• Should not be permitted across firewall boundaries (i.e. don’t extend the corporate domain into the SCADA DMZ)
• Exception – When a Read-only Domain Controller (drastically reducing port requirements) is used with an IPSec VPN tunnel connection to extend the SCADA domain into the SCADA DMZ
• Generally worth the trouble to ease implementation & maintenance of role based access & remote access using RADIUS authentication
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 30
Remote VPN Connections
• VPNs can securely extend WANs using public media & provide secure remote access to mobile staff
• Remote Facility Connections Using IPSec Site-to-Site VPNs• Used to interconnect two or more facility LANs
• Encrypts entire IP packet including endpoint private IP addresses
• Provides, confidentiality, data integrity, origin authentication and replay protection
• Mobile Remote User Connections TLS/SSL VPNs• Uses browser interface to connect mobile remote clients to servers
• Operate at the session level to provide secure client/server connections
• Uses certificates to authenticate servers & clients.
• Uses symmetric keys to provide confidentiality and data integrity
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 31
VPN Tunnel with Encryption
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA
Remote Access VPNs
32
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 33
Firewalls for Network Security and
Routing
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 34
Converged Plantwide Ethernet (CPwE)
Design & Implementation Guide (DIG)
• CPwE DIG – Developed by Cisco Systems & Rockwell Automation
• Provides detailed guidance & includes LAN configuration alternative testing results
• These figures from the DIG are from the LAN and DMZ design chapters
• LAN resilience alternatives shown & performance comparisons.
• L2&3 QoS settings recommendations
• DMZ Example tailored to SCADA
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 35
Designing and Implementation
Roadblocks
• Conflicts faced by utilities– Lack of regulatory driver
– Many competing needs
• Losing sleep each time another event makes the news.
• What to do? Utility staffs are a resourceful bunch and they find a way to address their concerns– Some are able to get funding to specifically address cyber security.
– Others have to be more creative.
• Utilities often lack resources to self-perform SCADA security assessments & improvements planning, design and implementation, as mentioned before, there is help.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 36
Designing and Implementation
Roadblocks, Continued
• Our experience– Some utilities, usually bigger ones, have adopted appropriate
standards and established internal policies, procedures and standards that they apply to each project
– More commonly, the utility hasn’t established comprehensive standards and isn’t aware of the vulnerabilities in their existing systems; but would like to make progress as part of each project.
– Sometimes it’s a grass roots or replacement project which means that they are open to a comprehensive solution but do have budgetary constraints.
– Other times it more like what can we shoehorn into this small incremental project.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 37
Example - Incremental Implementation
Well 10 Control Center3621 Redhill Place
Well 10Booster
PLC Box
Well 10ABuilding Control Panel
6Mb/s (CIR)
58/KDFN/103600/TWCS
20M
b/s
(CIR
)
58/K
DFN
/103
604/
TWC
S
10 BaseTX
100 BaseTX
Layer 3 Switch W/ EIGRP100
ASA 55102801/2611 W/ EIGRP100
Layer 2 Switch
TW TelecomMetro Ethernet
(ILAN)Layer-2 (Bridged)
Well8-SCADA-3000-1
WWTP1SCADA3 (HIST)
W-SCADA-WIN911
CityCenter-SCADA-3000-1
1536Kb/s (CIR)58/KDFN/103610/TWCS
Well15-SCADA-3000-1
W-SCADA-TS
WWTP12850 (EMAIL)
1536
Kb/
s (C
IR)
58/K
DFN
/103
674/
TWC
S
1536
Kb/s (C
IR)
58/K
DFN/103
608/TW
CS
1536Kb/s (CIR)
58/KDFN/103677/TWCS
Plant1-SCADA-2801-1
Plant2-SCADA-2801-1 Plant6-SCADA-2801-1
Plant6-SCADA-2960TT-1
Plant2-SCADA-2960TC-1 Fa0/0
Plant1-SCADA-2960TT-1
Plant 6 SCADA VIEW
WW-SCADA-SRVR1WW-SCADA-SRVR2
Well10-WATER-2960TC-1
W-SCADA-2960TT-1
Esteem
A
P B
ridge
Proxim
a A
P B
ridge
54mbps
Esteem AP Bridge
Exalt A
P
Bridge
Exalt A
P
Bridge
54mbps
Proxim
a A
P B
ridgeP
roxima
AP
Bridge
54mbps
Well10-WATER-2611-1
SCADA-ASA: 5510s in HA
W-SCADA-SERVER1 (Z400)
W-SCADA-CLIENT1 (WATER VIEW)
RX3I
RX3I
Remote VPN User
Remote VPN User
Remote VPN User
IPSec VPN Tunnel
100 BaseFX 62.5 125mu
PLC(Redundant Quantum)
WasteWater-SCADA-SERVER
PLC
MODBUS-IP-CONVERTER
WASTEWATER-SCADA-VIEW
Dell 2700
Well10-WATER-2955-1
OIT
PLC
OIT
PLC
PLC1A
PLC1B
HP JetDirect Lift Station 14.2
MTU PLC
P1Switch
SERVER
PLC
Switch
PLCPLC Cabinet
Tank 15
W_SCADA_CLIENT2
W DMZ
Esteem AP Bridge
OIU
WELL 15Westside Blvd SE
CITY CENTER
1536Kb/s (CIR)58/KDFN/103609/TWCS
WELL 8
WELL 10 (Control Center)
PLANT 2 Industrial Park
PLANT 1
Remote
Connections
Remote
Connections
Remote
Connections
Plant3-SCADA-2801-1
Plant3-SCADA-2960TT-1
MODBUS-IP-CONVERTER
PLANT 3Address?
1536Kb/s (CIR)58/KDFN/10367?/TWCS
Plant2-WW-L14_2-2955
1536Kb/s (CIR)58/KDFN/103608/TWCS Remote Connection to BPS 12
ATF
BPS
LS13 PLC
LS21 PLC
Mod
b us+
P3 PLC
LS 8
Esteem
A
ccess Point
54mbps
MODBUS-IP-CONVERTERLS8 PLC
Modbus Serial
Esteem
A
ccess Point
Modbus
54mbps
Esteem AP Bridge
Point-to-Point
Connection to Well 12
WW DMZ
W-SCADA-DEV-1
SHARED DMZ
[FUTURE]
SCADA-DMZ-RODC01(DC, NTP, Anti-virus, WSUS)
WW-SCADA-TS
WW-SCADA-WIN911
WW-SCADA-HIST
SCADA-DMZ-RODC02
TW Telecom
Internet
Well10CC-SCADA_DMZ-2960TT-1
WW-SCADA-CLNT
Well10CC-SCADA-3560-1
W-SCADA-2960TT-1
Tx/Fx
100BASE-TX/100BASE-FXMedia Converter
Proxim
a A
P B
ridge
SCADA-DCPRI
SCADA-DCSEC
SHAREDSCADA
MOSCAD MTU
Modbus
WW-SCADA-2960TT-1
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 38
Example – Incremental Installation
• Initial installation can be done
using a single Ethernet switch and
no remote connections.
• Remote connections can be added
in the future when they can be
secured correctly.
•Design supports adding disaster
recovery elements as budgets allow
• Initial equipment can be upgraded
in the future through firmware to add
required additional services such as
high availability.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 39
Example - Single Implementation by
Phased and Sequenced Construction
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 40
Keys to Successful Implementation
(Abbreviated Version of a Long List)
• Use equipment with a long useful lifetime and low risk of becoming completely obsolete in the short term.
• Have a budget in mind and idea of the risk/reward of network connected systems and equipment.
• Be aware that equipment cost is not an indication of work costs. A $1000 router could cost as much to configure as a $15,000 industrial router.
• Are staff or service contracts in place to maintain and troubleshoot systems? Systems are only as good as the maintenance done.
• Make sure that good system documentation and training will be delivered with the improvements.
• Set up a secure backup configuration storage mechanism & keep a copy of all addressing, configurations, settings, and software.
• Use qualified integrators having the proper certifications where appropriate.
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA
• A strategy for layering protection mechanisms to reduce the impact of a single mechanism failure
• In addition to the technical and operational controls that can be applied to SCADA systems, defense in depth requires long term organizational management and operations commitment to security for:– Developing security policies, procedures and educational
materials that apply directly to SCADA
– Conducting periodic security awareness, incident response and disaster recovery training
– Ongoing maintenance and upgrade of SCADA security throughout its lifecycle
– Restricting physical access to SCADA infrastructure
41
Defense in Depth
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 42
User Access
• Simple user interface.
• Do not allow access to the
start menu or other non-
essential programs
• Do not allow access to the
computer
• Require login credentials with
secure passwords and auto logouts
• Use USB security where ports are
available
2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 43
Summary
• The jury is in, the threat is real and utilities need to act
• Adequate guidance is available to support standards based cyber security improvements
• The DHS CSET tool and INL assessment support team provide a SCADA focused tool for conducting self-assessments
• Without a regulatory driver, funding continues to be a problem.
• Proper planning, implementation, and maintenance is key for a successful system. Systems cannot be installed and forgotten.
• Utilities are finding a way to make meaningful progress with both funding and solutions.