Cri$cal Infrastructure Power U$lity and SCADA Security
Kevin Manderson Hydro Tasmania
Hydro Tasmania
• Tasmania had the first hydro electric power sta$on in southern hemisphere at Duck Reach, Launceston;
• Australia's largest renewables generator and water manager;
• 30 power genera$ng sta$ons (hydro, gas and wind); • Dams, tunnels, weirs, flumes; and • Substan$al monitoring of infrastructure and environment.
Safety Moment
• October 7th 2013 – A TEPCO employee carelessly pressed a buUon shuVng off cooling pumps that serve the spent fuel pool in reactor #4 — thankfully a backup kicked in before any cri$cal consequences resulted.
• Process vulnerability analysed, managed and the outcome indicates it was appropriately controlled.
From my 1884 edi$on of ‘The Colonists Guide and Cyclopaedia’ The only men$on of ‘secure’ is securing a horse to a slippery pole – op$on, clove hitch
Another Dic$onary Defini$on
se·∙cu·∙ri·∙ty • noun \si-‐ˈkyu̇r-‐ə-‐tē\: the state of being protected or safe from harm
• : things done to make people or places safe
Summary
• Security could be described as ‘things that keep my systems from causing harm’,
• ‘Safe from harm’ depends on the nature of the system, • Jus$fying security is simply an NPV analysis of the consequence – but the likelihood moves,
• Implemen$ng good controls helps by having experience with opera$onal systems being aUacked, and
• My view is security is ‘mechanisms to stop changing the control I have over my systems and data to keep the systems from causing harm, without my knowledge’.
A security `something’ is what
• Any change which causes the overall process to be less safe or in an unknown state or safe,
• The specifica$on or expecta$on is not met, • So what can have an impact? – Malicious hacker, – A ‘Snowden’ or admin event, – Physical interven$on, – Negligence/Accident, – Equipment/sooware malfunc$on, and – Loss of services.
So What am I Keeping Secure
• The dispatch process: – Several sliding windows of $me with $ghtly coupled processes occurring across a group of closely and loosely coupled systems • Four seconds – data exchange with power sta$ons, • Eight seconds – control data exchange with AEMO, • Five minutes – dispatch interval, • 30 minutes – market interval, • Daily – repor$ng and other processes, • A number of ancillary services, and • Other contracted and mandated services.
• But am just the custodian of data in part of the chain of the complete process.
What Is Involved
• The core dispatch process is a con$nuum of: – receiving targets from AEMO, processing them, – sending targets to power sta$ons , and – receiving remote data and sending data to AEMO.
• The secondary dispatch processes are a con$nuum of receiving situa$onal events, local alarm processing/logging, managing water and other local risks
• Cut, rinse and repeat every four seconds, ad infinitum,
What Adds to the Mix
• A requirement for redundant systems,
• A requirement of 99.995% up$me,
• The need to get some data from corporate and other sources, and
• The need to pass data to corporate and others.
99.9% ("three nines")
8.76 hours
43.8 minutes
10.1 minutes
99.95% 4.38 hours
21.56 minutes
5.04 minutes
99.99% ("four nines")
52.56 minutes
4.32 minutes
1.01 minutes
99.999% ("five nines")
5.26 minutes
25.9 seconds
6.05 seconds
Year Month Day
Situa$on – What Bad End Game
• What could be the target – Aurora class event, – System black, – Dam failure, and – Or Just less safe…
What Touches my Systems
• The systems sit in locked racks, rela$vely inert, • What can have an impact? – People, – Data, – Services, and – Physical environment.
Considera$ons
• Cri$cal infrastructure SCADA model: – What core process is required, – Secondary processes/redundancy, and – Support and non essen$al systems.
• Tools: – Defence in depth, and – Vulnerability analysis and early security design.
• Start from the End Game: – The final last gasp must work, – Analyse processes, interac$ons and impact on security, and
– Can process layers be bypassed?
What Op$ons
• Air gap to corporate/internet. Doesn’t work, Nantanz/Iran – Stuxnet via USB key,
• Tightly lock down every point. Fails by social engineering, and
• Design in security from the start. Ongoing vulnerability analyses rela$ve to actual events, conferences and research -‐ then update security posture as needed. Best op$on.
Process Analysis
• Analyse to find the paths/vectors across the vulnerability layers,
• I think of each process/layer as a bubble inside or touching another bubble, and
• Consider controls for each layer or bubble: – If this bubble is popped will the system be resilient and the next bubble remain,
– Can the final ‘must keep intact’ bubble survive?
– Can any vulnerabili$es or layers be bypassed.
SCADA Good Prac$ce Guide (GPG)
Layered on GPG
Process Analysis
• Process 1 – Environment to support the core processes/must remain processes: -‐ Preserve ‘data chain’ for dispatch system -‐ AEMO to online generators and return.
• Process 2 – Redundancy and cri$cal support: -‐ Primary redundancy, communica$ons, compliance ‘func$onality’.
• Process 3 – Suppor$ng systems: -‐ Test, development, other support systems.
• Process 4 – Non or less essen$al services and systems: -‐ Historian, repor$ng, non opera$onal systems.
Hydro Architecture
• The GPG is a baseline, • Addi$onal $ers of access control, some as processes/layers, others as diversity in the boundary traversal and monitoring and aler$ng,
• Hydro’s produc$on environment has about 40 servers/systems integra$ng the produc$on process. Redundant over mul$ple sites and communica$ons paths. Builds on the GPG, and
• Logging, monitoring and aler$ng.
Process Approach
• Vulnerability analysis for each process flow, • Start with an assump$on that all external interfaces are or likely to be compromised (‘there be dragons’), and
• Brainstorm possibili$es: – People, – Services, – Black swans, and – All hazards approach – that is, all or none?
Deciding on Security Barriers
• Aoer analysis iden$fy the process change, ownership transfer and different security groups. These are highly likely to be the vulnerability points,
• What user group • What is the vulnerability: From AIC (or CIA):
– Availability: DOS/interrup$on, – Integrity: data or control injec$on/corrup$on, and – Confiden$ality: data access.
• What consequences, and • What cost to control (inputs or consequence).
Example -‐ Part of Cri$cal Path
• Basic func$onal requirements: – Cri$cal to process, data burst every 8 seconds, – ICCP (IEC 60870) traffic, – Bidirec$onal, high availability, and – To/from external provider.
Data to and from AEMO.
SCADA and Dispatch processing
Power sta$ons
Process -‐ Cri$cal Path
Data to and from AEMO
SCADA and Dispatch processing
Power sta$ons
• Func$onal requirements: – Cri$cal to process, data
burst every 8 seconds, – ICCP traffic, – Bidirec$onal, high
availability, and – To/from external provider.
• Controls implemented: – High availability cross
connected ‘enhanced firewalls’,
– Mul$ple firewalls/vendors, – Monitor link status, – Only allow ICCP traffic, and – Redundant paths.
Monitoring
What Else for Hydro
• Diversity of security barriers, • Golden images for switches,
• Scripted builds for install and then security, • ‘Whitelis$ng’. Hash lists of known and known bad and regularly compare, check updates,
• Monitoring (IDS, Full data packet capture, SMS alarms, environment), and
• Data valida$on.
Now Layer the ASD/DSD 35 List on Top
1. Applica$on whitelis$ng 2. Patch applica$ons 3. Patch opera$ng systems 4. Least privilege 5. Mul$ factor authen$ca$on 6. Firewalls 7. Segmented systems 8. IDS/IPS and monitoring 9. <down to 35>
ASD List
• Targeted at securing the IT components, • Notes: – In many SCADA or process control environments patching is poorly done,
– I know of one current opera$onal SCADA system running under Windows3.11, and
– I know many cri$cal infrastructure systems where patching is years old.
• But in SCADA environments generally everything else is far beUer than average.
ASD List
• Excellent for keeping individual systems secure,
• Many secure systems contribute to keeping an overall complex process secure, par$cularly at the less process intensive boundaries, and
• Consider where the IT component is important in the process chain.
Our Observed/Heard of Events
• Nil IT security • Hydro: – May 2005, CBD power fail caused transport/access issues, – Had a significant power failure event in April, and – Air-‐condi$oning event in June.
• Others: – Data from the field, range failures:
• SCADA input of 10^38 to another SCADA system , received ok but process failed with a squaring func$on.
– Peoples ac$ons, par$cularly when fa$gued: • October 2013, Tepco – operators pushed big red buUon, and • October 2013, loosened pipe and drenched in radioac$ve water.
Root Causes
• CBD power: Substa$on trip, alternate travel paths not considered,
• Power failure: Occurred during regular UPS tes$ng, system deigned to survive
• Air-‐condi$oning: Occurred during regular generator tes$ng, unexpected but survived (now monitored/controlled), and
• Data 10^38: no limit checking,
How to build a secure system
• Use the AG Good Prac$ce Guide as a guideline for the separa$on of form/func$on and privileges,
• Use the ASD 35 point security guide to help with ongoing hardening and maintenance procedures,
• Build for mul$ple concurrent failures. Expect mul$ple unrelated failures to occur simultaneously, and
• Think all hazards during analysis.
Think Alternate Security
• Who went to Ruxcon? • Other white/grey/black hat
conferences, • Use a range of tools and
test your systems, • Do pen tes$ng, • Think touch==own,
physical security is cri$cal, • Simple can be good, • Be aware of what's
happening, and • xkcd is good…
Comments or Ques$ons
