TowardsaSecureandResilientIndustrialControlSystemwithSo7ware-DefinedNetworking
Dong(Kevin)JinDepartmentofComputerScienceIllinoisInsEtuteofTechnology
TSS/SoSSeminar,March15,2016
1
Workwith…
3
WenxuanZhou JasonCro7 MaRhewCaesar BrightenGodfrey
ChristopherHannon JiaqiYan HuiLin ChenChen
JianhuiWang JunjianQi ZhiyiLi MohammadShahidehpour
Referencestopapersinthistalk
4
• WenxuanZhou,DongJin,JasonCro7,MaRhewCaesar,andBrightenGodfrey."EnforcingCustomizableConsistencyProperEesinSo7ware-DefinedNetworks."NSDI,2015
• ChristopherHannon,JiaqiYanandDongJin."DSSnet:AMicrogridModelingPlaeormwithElectricalPowerDistribuEonSystemSimulaEonandSo7wareDefinedNetworkingEmulaEon.”ACMSIGSIMPADS2016(toappear)
• HuiLin,ChenChen,JianhuiWang,JunjianQiandDongJin."Self-HealingARack-ResilientPMUNetworkforPowerSystemOperaEon."IEEETransacEonsonSmartGrid(submiRed)
• DongJin,ZhiyiLi,ChristopherHannon,ChenChen,JianhuiWang,MohammadShahidehpour."TowardsAResilientandSecureMicrogridUsingSo7ware-DefinedNetworking."IEEETransacEonsonSmartGrid,SpecialIssueonSmartGridCyber-PhysicalSecurity(submiRed)
IndustrialControlSystems(ICS)
5
• ControlmanycriEcalinfrastructures– e.g.,powergrids,gasandoildistribuEonnetworks,wastewatertreatment,transportaEonsystems…
• ModernICSesincreasinglyadoptInternettechnologytoboostcontrolefficiency,e.g.,smartgrid
NextGeneraEonofPowerGrid
LOADS SITESDISTRIBUTIONTRANSFORMER
DISTRIBUTIONSUBSTATION TRANSMISSION GENERATION
MoreEfficientorMoreVulnerable?
6 Picturesource:NISTFrameworkandRoadmapforSmartGridInteroperabilityStandards
Distribu(onOpsTransmission
Ops
OperaEons ServiceProviders
BulkGeneraEon DistribuEon Customer
MarketsRTO/ISOOps
DMS AssetMgmt
Enterprise
Bus
EMS
RTOSCADA
EMSWAMS
MDMSDemandResponse
Retailer/Wholesaler
Transmission
ISO/RTOParEcipaEon
Aggregator
EnergyMarketClearinghosue
MarketServicesInterface
PlantControlSystem
Generators SubstaEonDevice
FieldDevice
DistributedGeneraEon
U(lity
Provider
Third-Party
Provider
CIS
Billing
Home/BuildingManager
Aggregator
ElectricVehicle
DistributedGeneraEon
ElectricStorage
Appliances
ThermostatCustomer
EMS
CustomerEquipment
Meter
Others
CIS
Billing
RetailEnergyProvider
Premises
Networks
EnergyServicesInterface
MeteringSystem
DistribuEonSCADA
Enterprise
Bus
TransmissionSCADA
Enterprise
Bus
WideArea
Network
Substa(on
LANs
Internet/
e-business
FieldArea
NetworksData
Collector
SubstaEonController
ElectricStorage
Internet/
e-business
CommunicaEonPath Network
CyberThreatsinPowerGrids
7
Picturesource: 1.NaEonalCybersecurityandCommunicaEonsIntegraEonCenter(NCCIC).ICS-CERTMonitorSep2014–Feb20152.hRp://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-aRributed-hackers-take-down-power-grid/
• 245incidents,reportedbyICS-CERT
• 32%inenergysector
• 80,000residentsinwesternUkraine
• 6hours,lostpoweronDec23,2015
ProtecEonofIndustrialControlSystems
8
• Commercialof-the-shelfproducts– e.g.,firewalls,anEvirusso7ware– fine-grainedprotecEonatsingledeviceonly
• Howtochecksystem-widerequirements– Securitypolicy(e.g.,accesscontrol)– Performancerequirement(e.g.,end-to-enddelay)
• HowtosafelyincorporateexisEngnetworkingtechnologiesincontrolsysteminfrastructures?
ARepresentaEveSmartGridControlNetwork
9
ModernSubsta(onNetwork
SCADADataAggregator
WorkstaEonMonitor
...
relay breakerrelay breakerrelay breaker
RemoteTerminalUnit
RemoteTerminalUnit
modem
Real-Time
Control/
Monitoring
EnergyManagementSystem
DataHistorian
Time-criEcalControlUpdatesNetworkUpdates
ChallengesandOpportuni(es
DifferencesandSimilariEes
10
Similari(es
• blackholeavoidance• loopmiEgaEon• fastconvergencespeeds• prioritycontrol• mulEpleservicesonasingle
physicalchannel• …
AUElityControlNetwork AnEnterpriseNetwork
Differences
• strictlydefinedforwardingpaths• end-to-endperformance
guarantee• system-widevisualizaEon• real-Ememonitoring• adeny-by-defaultsecuritymodel• …
ProblemStatement
11
• MinimizethegapswithanSDN-enabledcommunicaEonarchitectureforICS
• CreateinnovaEveapplicaEonsforICSsecurityandresiliency– Real-EmenetworkverificaEon– Self-healingnetworkmanagement– Context-awareintrusiondetecEon– Manymore...
ICS–industrialcontrolsystemSDN–so7ware-definednetworking
12
Source:NickMcKeown,OpenNetworkingSummit2012
SpecializedControlPlane
SpecializedHardware
SpecializedFeatures
HorizontalOpeninterfacesRapidinnovaEon
VerEcallyintegratedClosed,proprietarySlowinnovaEon
AppAppAppAppAppAppAppApp
ControlPlane
ControlPlane
ControlPlane or or
OpenInterface
MerchantSwitchingChips
OpenInterface
SDNArchitecture
SDNArchitecture-ConEnue
Net3
Net4
Net5
Net6
Net1 Net2
OpenFlowProtocolDataPlane
ControlPlane
Applica0ons QoS AccessControl VPN
OpenFlowController
OpenFlowSwitches
13
CyberResources
SCADAServers
FieldDevices
CommunicaEonNetworks RouEng
PowerControlApplica(ons
DemandResponse
FrequencyControl
StateEsEmaEon
TopologyControl
…
…
• Instability• LossofLoad• SynchronizaEonFailure• ConEngency• LossofEconomics
Impact
DenialofService
FalseDataInjecEon Malware Insider
ARack…
CyberANacks
CurrentPowerGrid:PotenEalCyberARacksandTheirImplicaEons
FutureSDN-enabledPowerGrid:ACyber-ARack-ResilientPlaeorm
14
AnSDN-EnabledPowerGrid
PowerGridComponentLayer
PowerNetworkLayer
CommunicaEonNetworkLayer
SDNControlLayer
ApplicaEonLayer GridApplicaEon
ControlManagemen
tMonitoring
Commun
icaE
onSystems
Powergrid
Systems
SDNApplicaEon
IDSVerificaEon
Self-healingNetwork
SolarPV
GasGenerator
ChargingStaEon
WindTurbine
ComEdComEd
PershingSubstaEon(12.47kV)
FiskSubstaEon(12.47kV)
TransiEontoanSDN-EnabledIITMicrogrid• Real-EmereconfiguraEonofpowerdistribuEonassets• Real-EmeislandingofcriEcalloads• Real-EmeopEmizaEonofpowersupplyresources
15
ControlCenter
ExisEngMasterController
SDNMasterController
SDNApplica0ons
GridApplica0onsLocalSDNController1
PMU
LocalSDNController2BuildingControl
LocalSDNControllern
…
CommunicaEonNetworks
SolarPV
GasGenerator
ChargingStaEon
WindTurbine
ComEdComEd
PershingSubstaEon(12.47kV)
FiskSubstaEon(12.47kV)
16
TransiEontoanSDN-EnabledMicrogrid
• SDN-basedApplicaEons– Real-EmeVerificaEon– Self-healingPMU
• HybridTestbed– SDNemulaEon+PowerDistribuEonSystemSimulaEon
17
ApplicaEon1:NetworkVerificaEon–MoEvaEon
18
• Unauthorizedaccess• UnavailablecriEcalservices• Systemperformancedrop• Instability• Lossofload• SynchronizaEonFailure
• …
89%ofoperatorsneversurethatconfigchangesarebug-free1
82%concernedthatchangeswouldcauseproblemswithexisEngfuncEonality1
1. Surveyofnetworkoperators:[Kim,Reich,Gupta,Shahbaz,Feamster,Clark,USENIXNSDI2015]2. PicturesborrowedfromVeriFlowslides[Khurshid,Zou,Zhou,Caesar,GodfreyNSDI2013]
VerificaEonSystemDesign
19
ICSApplica(onModels
NetworkModels
PolicyEngine
topologynetwork-layerstates
(e.g.,forwardingtables)
Diagnosis
• Vulnerabili0es• Errors
SystemFramework
DynamicModelUpdate/Selec3on Verifica3on
DynamicNetworkData(topology,forwardingtables…)DynamicApplicaEonData(controlupdates…)User-specifiedPolicy(security,performance…)
VerifiedSystemUpdates
VeriFlow
New rules
VeriFlow Operation
4/3/2013 Department of Computer Science, UIUC 11
Network Controller
Generate equivalence
classes
Generate forwarding
graphsRun queries
Diagnosis report• Type of invariant
violation• Affected set of
packets
Rules violating network invariant(s)
Good rules
Network-LayerVerificaEon
20
PriorWork• FlowChecker
[Al-Shaeretal.,SafeConfig2010]• HeaderSpaceAnalysis
[Kazemianetal.,NSDI2012]• Anteater
[Maietal.,SIGCOMM2011]• VeriFlow
[Khurshidetal.,NSDI2012]
PicturesborrowedfromVeriFlowslides[Khurshid,Zou,Zhou,Caesar,GodfreyNSDI2013]
21
Switch'A' Switch'B'
Controller'Remove&rule&1& Install'rule'2'
rule%1%
rule%2%
Challenges—TimingUncertaintyNetworkdevicesareasynchronousanddistributedinnature
Packet'
Challenges—TimingUncertainty
22
Switch'A' Switch'B'
Controller'
Install'rule'2'
rule%1%
rule%2%
Remove&rule&1&(delayed)&
Loop-freedomViolaEon
Uncertainty-awareModeling• Naively,representeverypossiblenetworkstateO(2^n)• Uncertaingraph:representallpossiblecombinaEons
23
UpdatesynthesisviaverificaEon
EnforcingdynamiccorrectnesswithheurisEcallymaximizedparallelism24
AshouldreachB
2 1 3 4
SlideborrowedfromWenxuanZhou,“CCG”NSDI2015
OK,but…
25
Canthesystem“deadlock”?• Provedclassesofnetworksthatneverdeadlock• ExperimentallyrareinpracEce!• Lastresort:heavyweight“fallback”likeconsistentupdates[ReitblaRetal,SIGCOMM2012]
Isitfast?
0
5000
10000
15000
20000
25000
0 2 4 6 8 10 12 14 16
25000$
20000$
15000$
10000$
5000$
0$7/22/2014$22:00:00$
7/22/2014$23:00:00$
7/23/2014$0:00:00$
7/23/2014$1:00:00$
//$
//$
//$
//$
//$
//$
Time$
Num
ber$o
f$Rules$
in$th
e$Network$
7/22/2014$22:00:02$
7/22/2014$23:00:02$
7/23/2014$0:00:02$
7/23/2014$1:00:02$
0
5000
10000
15000
20000
25000
0 2 4 6 8 10 12 14 16
Immediate Update
GCC
Consistent Updates 0
5000
10000
15000
20000
25000
0 2 4 6 8 10 12 14 16
Immediate Update
GCC
Consistent Updates
End
End
End
Comple?on$Time$} CCG
0
5000
10000
15000
20000
25000
0 2 4 6 8 10 12 14 16
Immediate Update
GCC
Consistent Updates
End
End
End
0
5000
10000
15000
20000
25000
0 2 4 6 8 10 12 14 16
Immediate Update
GCC
Consistent Updates
End
End
End
SlideborrowedfromBrightenGodfrey,TSSSeminar,Sep2015
ApplicaEon2:Self-HealingPhasorMeasurementUnit(PMU)Networks
26
IntegraEonofACommunicaEonNetworkandAPMUNetwork
Self-HealingPMUNetworks
27
VideoDemo
• Isolatecompromiseddevices
• “Self-heal”thenetworkbyquickly
re-establishingroutes
– Torestorepowersystemobservability
– Usinganintegerlinearprogrammodel
AHybridTesEngPlaeorm
28
Windows Linux
Power Coordinator● Setup Simulator ● Communicates Requests
between Emulator and Simulator
COM Port
Network &
IEDConfiguration
Network Coordinator● Configure Network
and Hosts● Synchronize with
Simulator
Synchronization Events
zmq socketKernel
Input or Import
Named Pipe
TCP Socket
Windows COM Port
Legend
DSSnetConfiguration
Processes/Elements
Components
Virtual Time System
IED Configuration
Power Element Configuration Mininet
HOSTS SWITCHES
CONTROLLER
Settings
OpenDSSElementsElements
MonitorsMonitors Controls
Circuit
Interface
Figure 2: DSSnet system architecture diagram. Note that the power simulator runs on a Windows machine and the networkemulator runs on a Linux machine.
to advance the simulation’s clock to the time stamp of thecurrent event request and to solve the power flow at thattime. Additionally, some elements of the power grid maybe modeled in the power coordinator as a function of time,such as loads and generation. These elements are not nec-essarily represented in the communication network, but canstill operate on DSSnet’s virtual clock.
3.1.5 Virtual Time System
Unlike simulation, the emulation clock elapses with thereal wall clock. Therefore, pausing the emulation requiresmore than just stopping the execution of the emulated enti-ties, but also pausing their clocks. Virtual time can be usedto achieve this goal [9, 19]. We choose to extend the workof [9], in which Mininet is patched with virtual time support.However, their motivation is di↵erent from ours.
In general, virtual time has at least two categories of ap-plication. The first one is to slow down emulation so thatit appears to emulated entities that they have su�cient vir-tual resources. Slowing down execution also alleviates theproblems caused by resource multiplexing. Another usage ofvirtual time is for emulation-simulation synchronization. InDSSnet, we assign every container a private clock, insteadof using the global time provided by the Linux OS. The con-tainers now have the flexibility to slow down, speed up orstop its own clock when synchronizing with the simulator.
However, the emulator needs to manage the consistencyacross all containers. This is achieved by a centralized time-keeper in [19], and by a two-layer consistency mechanism [9].In practice, the emulator configuration guarantees that all
containers are running with one shared virtual clock; Simi-larly, the container leverages the Linux process hierarchy toguarantee that all the applications inside the container areusing the same virtual clock. The two-layer consistency ap-proach is well-suited to this work for pausing and resumingbecause:
1. All hosts should be paused or resumed when we stopor restart the emulation.
2. All processes inside a container should be paused orresumed when we stop or restart the emulation.
The first task is done by the network coordinator. The sec-ond task is implemented based on the fact that processesinside a container belong to the same process group.
3.2 SynchronizationA key challenge in DSSnet is the synchronization between
connecting the emulated communication network and thesimulated power system. The root cause is that two di↵er-ent clock systems are used to advance experiments. Ordi-nary virtual-machine-based network emulators use the sys-tem clock, and a simulator often uses its own virtual clock.This di↵erence would lead to causality errors as shown inthe following example.In Figure 3, there are three cross-system events (E
i
), eachwith a response (R
i
). E1 occurs before E2, however, E2 mayrequire information from R1. Since the response occurs afterthe second event, the global causality is violated, and thusreduces experiment fidelity. An example of E1 is a request
PowerDistribuEonSystemSimulaEon+SDN-basedNetworkEmulaEon
AHybridTesEngPlaeorm
29
• Challenges– TemporalfidelityinnetworkemulaEon– SynchronizaEonbetweentwosub-systems• EmulaEon–execuEng“naEve”so7waretoproducebehaviorinwall-clockEme• SimulaEon–execuEngmodelso7waretoproducebehaviorinvirtualEme
Ourapproach:VirtualTime
• Keyidea:tradeexecuEonEmewithfidelity• TimedilaEonfactor(TDF)[Gupta,2011]
• TDF=10– 10secondsinrealEme<=>1secondinaEme-dilatedemulatedhost
– a100Mbpslinkisscaledtoa1Gbpslink
D.Gupta,K.V.Vishwanath,etal.“Diecast:TesEngdistributedsystemswithanaccuratescalemodel”.ACMTransacEonsonComputerSystems,29(2):1–48,2011
36
VirtualTimeSystemArchitectureforaContainer-basedNetworkEmulator
Virtual Time ClockOpen vSwitch
Virtual Network
Virtual Time Keeping
……LinuxContainer
TimeDilation
LinuxContainer
LinuxContainer
Time Freezer
Freezer Command Flow
Dilation Command Flow
Sourcecode:hRps://github.com/liRlepreRy/VirtualTimeForMininet
37
FutureWork
39
• MoreapplicaEons– e.g.,SpecificaEon-basedIntrusionDetecEon
• NetworklayeràApplicaEonlayerandCross-layerverificaEon
• In-houseresearchideaàRealsystemdeployment– IITMicrogrid– FirstClusterofMicrogridsinUS(12MWIIT+10MWBronzeville)
VirtualizedU(lityNetwork1
FrequencyControlVirtualizedU(lityNetwork2
DemandResponse
VirtualizedU(lityNetwork3
StateEsEmaEonVirtualizedU(lityNetwork4
TopologyControl
ControlCenter
Cross-Layer
Verifica(on
Intrusion
Detec(on
40
SpecificaEon-basedIntrusionDetecEon
Cross-layerVerificaEon
41
CommunicaEonNetworklayer
PowerControlApplicaEonlayer
AnetworkenvironmentwithdesiredproperEes(performance,security…)
Correctappbehaviors