38
8 th Security Summit Portland, Oregon Putting SCADA Security to the Test: Why you need a lab and how to get one Chris Sistrunk, PE Sr. Engineer Entergy – Jackson, MS
8th Security Summit Portland, Oregon
Putting SCADA Security to the Test: Why you need a lab and how to get one
Chris Sistrunk, PE Sr. Engineer Entergy – Jackson, MS
8th Security Summit Portland, Oregon
Why do we need a lab,
Chris?
8th Security Summit Portland, Oregon
What happens when you use nmap on an Industrial Control System
http://securityreactions.tumblr.com
8th Security Summit Portland, Oregon
Why do we need a lab?
With a lab, you can
• Test relay and RTU settings on a replica of production systems
• Test new firmware before issuing to field
• Perform root-cause analysis
– Why is this device locking up once a month?
• Try out new equipment from a vendor
8th Security Summit Portland, Oregon
Why do we need a lab?
Save time & money by
• Creating standard settings templates
• Find problems before they are widespread
(Not having to recall units with firmware issues)
• Develop and test equipment pilots in-house rather than hiring a company to do it
• Use lab equipment as emergency spare
8th Security Summit Portland, Oregon
Why security testing?
• Not all SCADA/relay vendors do negative or security testing at their factories
• Even if they did, they can’t test equipment the EXACT way that you use it
• Test your own equipment before hackers or some drive-by malware does it for you
• Use the results to mitigate vulnerabilities
8th Security Summit Portland, Oregon
What kinds of testing?
8th Security Summit Portland, Oregon
• Factory/Site Acceptance Testing (RTU system)
• Firmware/Software Testing (new or patches)
• Protocol Testing (DNP3, Modbus, etc)
• Protocol Fuzzing (custom or off-the-shelf)
• Penetration Testing (Metasploit, etc)
• Physical security testing (cabinet locks etc)
• DOCUMENT! DOCUMENT! DOCUMENT!
What kinds of testing?
8th Security Summit Portland, Oregon
What would be your stuxnet?
• Be a hardhat hacker
• Think like an attacker who has your prints!
• Build your systems with layers of defense
• If you find a vulnerability, let your vendor know (they might even have a patch)
“To make things work well, you must break them!”
8th Security Summit Portland, Oregon
How I Audit SCADA Systems
http://securityreactions.tumblr.com
8th Security Summit Portland, Oregon
OK, how do I get a lab?
8th Security Summit Portland, Oregon
OK, how do I get a lab?
• Ask your boss! Ask the CIO! Ask Ask Ask!
• If you are the boss, ask your best people what they want in their lab and go buy it!
• Put together a plan or a business case! – Add it to NERC/CIP compliance budget (big driver)
• Go get spare equipment and make a rack!
• Start small and add to it. – Mine started as 2 relay racks in my cubicle
8th Security Summit Portland, Oregon
Some ideas
8th Security Summit Portland, Oregon
Still can’t afford it?
8th Security Summit Portland, Oregon
Can’t afford one, don’t have the manpower, don’t have the expertise?
• 3rd party testing such as Enernex, Digital Bond, Kinectrics, Cimation to name a few
• The US Gov’t has the Idaho NL National SCADA Test Bed, Pacific NW NL, & Sandia NL
• Colleges such as Louisiana Tech, Mississippi State, Jackson State have power, SCADA, and security equipment in their labs
• Farm out the testing and work with them to get the results you want & capitalize the test costs
8th Security Summit Portland, Oregon
“Engineering isn't about perfect solutions; it's about doing the best you can with limited resources.” -Randy Pausch, The Last Lecture
Engineering Truth
8th Security Summit Portland, Oregon
To be the best, you need the best tools!
8th Security Summit Portland, Oregon
Entergy THQ Virtual Lab Tour
8th Security Summit Portland, Oregon
Transmission HQ Labs
• Transmission HQ moved from NOLA to Jackson
• Business continuity after Hurricane Katrina
• Brand new building in Fall of 2009
• 5 large rooms designated for lab space – Relay & SCADA Lab
– Communications & Security Lab
– Real-time Power System Simulator Lab
– Mississippi Grid Lab
– High Voltage Lab
8th Security Summit Portland, Oregon
Relay & SCADA Lab
8th Security Summit Portland, Oregon
Relay & SCADA Lab NO
LAB RATS OR
CYBERATTACK SQUIRRELS ALLOWED
8th Security Summit Portland, Oregon
Relay & SCADA Lab
8th Security Summit Portland, Oregon
Relay & SCADA Lab
• Cubicle: 2 racks >> Old Break Room: 7 racks
• New THQ: 15 bolted racks, 10 rolling racks – 40+ Protective Relays (7 different standard panels)
– Digital Fault Recorder
– 8+ RTUs, 3 Communication Processors
– Substation Grade LAN & Corp Network
– GPS Clock (IRIG-B), HMI Screen & Keyboard
– Toolbox, O-Scope, Multimeter, Cables, Workstations, Chip Burner, Relay & RTU Test Sets, etc
8th Security Summit Portland, Oregon
Relay & SCADA Lab
• THE LAB OF MY DREAMS!
• We can replicate almost any substation
• Test new configurations
• Test problematic field configurations
• Test new firmware & software
• Test drive new equipment
• Train relay & RTU technicians and engineers
8th Security Summit Portland, Oregon
Communications & Security Lab
8th Security Summit Portland, Oregon
Communications & Security Lab
• Substation Hardened Router & Switch
• Radios of different bands and technologies
• Six-sided PSP for simulating CCA sites
• Several field firewalls
• Wurldtech Achilles Fuzzer – Test network robustness of devices
– Fuzzing DNP3, Modbus, & IEC 61850
– Test new RTU & Relay firmware patches
– Will network storm affect control outputs?
8th Security Summit Portland, Oregon
Power Real-Time Simulator Lab
8th Security Summit Portland, Oregon
Power Real-Time Simulator Lab
8th Security Summit Portland, Oregon
Power Real-Time Simulator Lab
“Hypersim is the only real-time digital simulator with the power to simulate and analyze very large-scale power systems with more than 2000 three-phase buses.” - http://www.opal-rt.com
• Simulate different fault scenarios
– Will the Relay A, B, C have a misoperation?
– Will relay fault activity affect comm (vice versa)?
• R&D & commissioning tests
8th Security Summit Portland, Oregon
Mississippi Grid Lab
8th Security Summit Portland, Oregon
Mississippi Grid Lab
• Multipurpose type lab used by Entergy Mississippi T&D Grid Engineers
• Inspecting/repairing equipment
• Pre-test new panels before field installation
• Spare parts inventory
8th Security Summit Portland, Oregon
High Voltage Lab
8th Security Summit Portland, Oregon
High Voltage Lab
• The Hi-VARC (High Voltage AC Resistive Current) test set provides rapid, automatic evaluation of MOV arresters and polymer insulators using AC voltages up to 132kV.” http://www.jmxservices.com
• Inspection & root cause of failed insulators, HV circuit breaker components, etc
8th Security Summit Portland, Oregon
Last but not least…
8th Security Summit Portland, Oregon
Go make stuff…Go break stuff
8th Security Summit Portland, Oregon
A Few Thoughts
SCADA Security isn’t easy
• Doing the best we can with what we have
SCADA, Relay, & Security Labs
• Having a lab is so valuable for testing, troubleshooting, breaking & fixing stuff
• Yes I have a fuzzer and I’m not afraid to use it
DNP3/IP Secure Authentication v5
• Please tell your vendors you want NEED it
8th Security Summit Portland, Oregon
Dream BIG!
8th Security Summit Portland, Oregon
Follow @chrissistrunk
Questions?
