Publication doi Preprint doi *correspondence: [email protected]Design and Evaluation of ACyber-Physical Resilient Power System Testbed Preprint , compiled November 30, 2020 Abhijeet Sahu 1 , Patrick Wlazlo 2 , Zeyu Mao 1 , Hao Huang 1 , Ana Goulart 2 , Katherine Davis 1 , and Saman Zonouz 3 1 Electrical and Computer Engineering, Texas A&M University, College Station, US 2 Electronic Systems Engineering Technology, Texas A&M University, College Station, US 3 Electrical and Computer Engineering, Rutgers University, New Jersey, US Abstract A power system is a complex cyber-physical system whose security is critical to its function. A major challenge is to model and analyze its communication pathways with respect to cyber threats. To achieve this, the design and evaluation of a cyber-physical power system (CPPS) testbed called Resilient Energy Systems Lab (RESLab) is presented that captures realistic cyber, physical, and protection system features. RESLab is architected to be a fundamental tool for studying and improving the resilience of complex CPPS to cyber threats. The cyber network is emulated using Common Open Research Emulator (CORE) that acts as a gateway for the physical and protection devices to communicate. The physical grid is simulated in the dynamic time frame using PowerWorld Dynamic Studio (PWDS). The protection components are modeled with both PWDS and physical devices including the SEL Real-Time Automation Controller (RTAC). Distributed Network Protocol 3 (DNP3) is used to monitor and control the grid. Then, exemplifying the design and validation of these tools, this paper presents four case studies on cyber-attack and defense using RESLab, where we demonstrate false data and command injection using Man-in-the-Middle and Denial of Service attacks and validate them on a large-scale synthetic electric grid. 1 Introduction The electric grid is transitioning to a smarter grid that employs advanced communication technologies. With advanced com- puting and communications, cyber-security has proven to be a critical issue in power transmission, generation, and distribu- tion systems. Cyber adversaries can modify or create data that can impact the grid’s normal operation and potentially destabi- lize its operating point causing cascading failures. Earlier this year, an unidentified threat successfully compromised the ad- ministrative systems of the European Network of Transmission System Operators for Electricity (ENTSO-E), with the poten- tial to compromise 42 transmission system operators (TSOs) across 35 member states in Europe [1]. Other attacks are also widely known such as the Ukraine attacks [2], where an attacker targeted three distribution units to cause a power outage after intruding into the Supervisory Control and Data Acquisition (SCADA) system. Attacks like Pivnichna [3] caused a power outage, while Stuxnet [4] allowed control of programmable logic controllers (PLCs), by overspeeding the centrifuges in a nuclear plant. It is necessary to propose defense mechanisms for such zero-day attacks. The use of firewalls, intrusion detection systems, and intrusion prevention systems is important, but these tools may not work efficiently on stealthy coordinated attacks. Hence, we need to employ the latest tools and techniques to make solu- tions that are more intelligent and capable of detecting complex attacks. Machine learning, including deep learning, or even artificial intelligence, offer advantages that can aid cyber and physical attack detection and localization. These techniques are data-intensive, where more data typically provides a better solution. One way to generate those real-time data sets is to mimic those attacks and defense mechanisms using a testbed. This paper presents our Resilient Energy Systems Laboratory (RESLab) testbed that forms an environment for researchers and stakeholders to understand the impact of cyber-attacks and validate their defenses. It provides a platform to evaluate how the power and communication networks perform together based on real-world systems and events, including communication protocols, operations, and latency requirements. It allows other researchers to develop and test intrusion detection tools for defending and mitigating real cyber attacks. These are the major contributions of this paper: 1. To introduce RESLab, a cyber-physical power system testbed that is designed to study resilience problems and solutions in large-scale power systems RESLab is a mix of emulators, simulators, and real devices that allow us to evaluate multi-stage cyber threats to the power system. 2. To model realistic data flows in the RESLab testbed using a large-scale exemplar power system based on utility architecture. This enables us to implement and validate scenarios impacting grid resilience such as false data and command injection uses cases. 3. To compare RESLab with other testbeds and present how RESLab is able to implement and validate realistic use cases for grid cyber-resilience. 4. To implement Denial of Service (DoS) and Man-in-the- Middle (MiTM) attacks and validate them by studying their impacts on the normal power system operation. 5. To provide a platform for data collection and visualiza- tion by integrating monitoring tools such as Packetbeat and Zabbix, Snort for intrusion detection, and a cyber- physical energy management system application. arXiv:2011.13552v1 [eess.SY] 27 Nov 2020
Design and Evaluation of A Cyber-Physical Resilient Power SystemTestbed
Preprint, compiled November 30, 2020
Abhijeet Sahu1, Patrick Wlazlo2, Zeyu Mao1, Hao Huang1, Ana Goulart2, Katherine Davis1, and Saman Zonouz3
1Electrical and Computer Engineering, Texas A&M University, College Station, US2 Electronic Systems Engineering Technology, Texas A&M University, College Station, US
3Electrical and Computer Engineering, Rutgers University, New Jersey, US
AbstractA power system is a complex cyber-physical system whose security is critical to its function. A major challengeis to model and analyze its communication pathways with respect to cyber threats. To achieve this, the designand evaluation of a cyber-physical power system (CPPS) testbed called Resilient Energy Systems Lab (RESLab)is presented that captures realistic cyber, physical, and protection system features. RESLab is architected tobe a fundamental tool for studying and improving the resilience of complex CPPS to cyber threats. The cybernetwork is emulated using Common Open Research Emulator (CORE) that acts as a gateway for the physical andprotection devices to communicate. The physical grid is simulated in the dynamic time frame using PowerWorldDynamic Studio (PWDS). The protection components are modeled with both PWDS and physical devicesincluding the SEL Real-Time Automation Controller (RTAC). Distributed Network Protocol 3 (DNP3) is usedto monitor and control the grid. Then, exemplifying the design and validation of these tools, this paper presentsfour case studies on cyber-attack and defense using RESLab, where we demonstrate false data and commandinjection using Man-in-the-Middle and Denial of Service attacks and validate them on a large-scale syntheticelectric grid.
1 Introduction
The electric grid is transitioning to a smarter grid that employsadvanced communication technologies. With advanced com-puting and communications, cyber-security has proven to be acritical issue in power transmission, generation, and distribu-tion systems. Cyber adversaries can modify or create data thatcan impact the grid’s normal operation and potentially destabi-lize its operating point causing cascading failures. Earlier thisyear, an unidentified threat successfully compromised the ad-ministrative systems of the European Network of TransmissionSystem Operators for Electricity (ENTSO-E), with the poten-tial to compromise 42 transmission system operators (TSOs)across 35 member states in Europe [1]. Other attacks are alsowidely known such as the Ukraine attacks [2], where an attackertargeted three distribution units to cause a power outage afterintruding into the Supervisory Control and Data Acquisition(SCADA) system. Attacks like Pivnichna [3] caused a poweroutage, while Stuxnet [4] allowed control of programmable logiccontrollers (PLCs), by overspeeding the centrifuges in a nuclearplant.
It is necessary to propose defense mechanisms for such zero-dayattacks. The use of firewalls, intrusion detection systems, andintrusion prevention systems is important, but these tools maynot work efficiently on stealthy coordinated attacks. Hence, weneed to employ the latest tools and techniques to make solu-tions that are more intelligent and capable of detecting complexattacks. Machine learning, including deep learning, or evenartificial intelligence, offer advantages that can aid cyber andphysical attack detection and localization. These techniquesare data-intensive, where more data typically provides a bettersolution. One way to generate those real-time data sets is tomimic those attacks and defense mechanisms using a testbed.
This paper presents our Resilient Energy Systems Laboratory(RESLab) testbed that forms an environment for researchersand stakeholders to understand the impact of cyber-attacks andvalidate their defenses. It provides a platform to evaluate howthe power and communication networks perform together basedon real-world systems and events, including communicationprotocols, operations, and latency requirements. It allows otherresearchers to develop and test intrusion detection tools fordefending and mitigating real cyber attacks.
These are the major contributions of this paper:
1. To introduce RESLab, a cyber-physical power systemtestbed that is designed to study resilience problemsand solutions in large-scale power systems RESLab isa mix of emulators, simulators, and real devices thatallow us to evaluate multi-stage cyber threats to thepower system.
2. To model realistic data flows in the RESLab testbedusing a large-scale exemplar power system based onutility architecture. This enables us to implement andvalidate scenarios impacting grid resilience such asfalse data and command injection uses cases.
3. To compare RESLab with other testbeds and presenthow RESLab is able to implement and validate realisticuse cases for grid cyber-resilience.
4. To implement Denial of Service (DoS) and Man-in-the-Middle (MiTM) attacks and validate them by studyingtheir impacts on the normal power system operation.
5. To provide a platform for data collection and visualiza-tion by integrating monitoring tools such as Packetbeatand Zabbix, Snort for intrusion detection, and a cyber-physical energy management system application.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 2
This paper is organized as follows. In Section 2, we evaluatetestbeds that model a cyber-physical power system and allowreal-time experiments. Section 3 presents the architecture andcomponents of RESLab. The threat model for this work is pre-sented in Section 4. Section 5 demonstrates its implementationin RESLab with four use cases. Then, we present the analysis ofthe attacks and their impacts on the physical system. The resultsare analyzed in Section 6.
2 Cyber-Physical Power System Testbeds
In this section, we first review testbeds that focus on investigat-ing vulnerability of power critical infrastructure, including theirchallenges and limitations. Previous works range from applica-tions in wide-area protection and monitoring in transmission andgeneration, to distributed energy resources (DERs), to micro-grids and distribution systems, and to operation domains such asEnergy Management Systems (EMS) and Distribution Manage-ment Systems (DMS). The comparisons in this section considerease of deployment and troubleshooting, design complexity, andcost of implementation. Then, we motivate RESLab, outliningits contributions, unique features, and how it fills existing gaps.
2.1 Testbeds and platforms
A cyber-physical testbed architecture is implemented by net-working together simulators, emulators, and hardware. Thequality of a cyber-physical testbed is measured by its successin advancing the research and applications that it supports. Itincludes a platform enabling communication between compo-nents, a system for data collection, aggregation, visualization,and a way of executing and evaluating cyber security incidentsagainst the system under study.
We present all the reviewed testbeds in Table 1 based on thepower and cyber simulators, communication protocols, softwareand devices, system level, intrusions type, and the application ofDERs.
2.2 Network representation
Various testbeds such as [5, 6, 7, 8, 9, 10] focus on evaluat-ing impact for physical use cases (e.g., cyber-attacks on powerflows, loss of load or synchronism, protection systems, andmicrogrids) while using networking hardware. Other testbeds[11, 12, 13, 14, 15] use network simulators, but the networksimulator’s primarily focus is on communication algorithmssuch as congestion control or bandwidth allocation schemes.For performing cyber-physical studies in a testbed, networkemulation is the preferred alternative to simulation. While asimulator demonstrates the behavior of a network, an emulatorfunctionally replicates the behavior of the network. Thoughsome simulators provide features such as System-in-The Loop(SITL) [11, 16] and Tap bridge [15] to integrate external devicesor virtual machines (VMs), those features are not scalable tolarge systems.
Virtualization and emulation enable scalability. In [17], thevSphere ESXi virtualization environment is used to simulatenetwork-based attacks; however, these scenarios are not specificto SCADA and focus on web browsing and file transfers. ThevSphere environment is reset before each simulation test run;
this provides built-in mitigation against any damage resultingfrom an attack simulations.
The selection of a specific platform requires design decisions thatare based on trade-offs in cost and accuracy. Network devicessuch as firewalls and routers can be included in the design,but are expensive for a large scale CPS. Network simulatorssuch as Opnet, Omnet, and Network Simulator-3 (NS-3) canbe cost-effective, but they do not provide a platform for real-time data processing of industrial protocols such as IEC 61850,Distributed Network Protocol version 3 (DNP3), inter-controlcenter communications protocol (ICCP), and Modbus.
The Common Open Research Emulator (CORE) provides a plat-form to run different applications, such as iptables for firewall,Snort for intrusion detection, and services such as Secure Shell(SSH) for remote access. CORE is used for emulating smart gridnetworks in [18], where the authors compare existing works ofco-simulation and discover CORE to be suitable for large-scalesimulations. Similarly, in [19] an Army microgrid is simulated,and its communication network is emulated with CORE, whereresults demonstrate the ability to implement cyber intrusions andevaluate the impact on the microgrid. However, testbeds thatuse CORE have not yet integrated network monitoring tools.
Sandia National Laboratories (SNL) offers a tool for launchingand managing virtual machines (VMs) to emulate large scalenetworks [20] called Minimega. It supports Virtual Local AreaNetworks (VLANs) with configurable bandwidth and qualityof service (QoS); thus, it can be used with a router’s operatingsystem and Linux kernels to emulate a communication network.Additionally, a power system and cyber co-simulation environ-ment called SCEPTRE [21] is being developed by SNL to allowhigh-fidelity simulation of SCADA protocols with hardware-in-the-loop such as PLCs and remote terminal units (RTUs), usingMinimega to manage the VMs.
A Hierarchical Engine for Large-scale Infrastructure Co-Simulation (HELICS) is developed by National Renewable En-ergy Laboratory for large-scale co-simulation, using off-the-selfpower system, and communication markets [22]. This frame-work integrates discrete-event simulators, such as NS-3, andtime-series simulations such as for power flows.
2.3 Power system representation
Power system simulators such as real-time digital simula-tor (RTDS), OpalRT, or Typhoon have been used in severaltestbeds [23, 24, 25, 11, 7, 6, 26, 27, 8, 10, 28]. These expen-sive hardware solutions are essential for experiments on electro-magnetic transients or power electronics, DERS, or microgrids.PowerWorld Simulator and Dynamic Studio (PWDS) provide so-lutions for large-scale power system modeling in the steady stateand transient stability time frames [29, 30, 31]. The testbeds thatuse hardware devices such as SEL relays, phasor measurementunits (PMUs), or RTACs for their experiments face challengesin understanding vendor-specific industrial control system (ICS)protocols. The testbed solutions in [23, 24, 25, 32, 9, 27]use physical hardware and consider ICS protocols such as IEC61850, C37.118 in their experiments.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 3
2.4 Operational technology protocols
Widely-known tools in information technology (IT) security,such as Ettercap and Metasploit frameworks, are not tailored tooperational technology (OT) or attacks to power systems. Forexample, a MiTM attack would need to be specific to SCADAprotocols, and representing such attacks is imperative for de-fense. Incorporating attacks and defense into a testbed requiresknowledge of the protocols as it involves inspecting and modify-ing packets. Vendor-specific protocols that are not open sourceare challenging to incorporate and evaluate in such environ-ments.
2.5 Constraints on use cases
Existing testbeds often lack strong demonstrations of cyber intru-sions. Several existing testbeds [23, 32, 33, 34, 35, 12, 36, 37]mention implementation of cyber intrusions in their platformsbut do not clearly demonstrate specific use cases including howthe intrusions are performed. Some testbeds that use MiTMattacks [25, 7, 26, 6, 8] do not show how those attacks areincorporated. For example, a MiTM attack can be performedin different ways such as address resolution protocol (ARP)cache poisoning, internet protocol (IP) spoofing, or hypertexttransfer protocol (HTTP) session hijacking. Some testbeds suchas [11, 38] use Ettercap or Metasploit frameworks but havelimitations in carrying out goal-oriented MiTM cyber-attacks.
Realistic use case support is a major feature of a testbed. For ex-ample, extensive research has been proposed on defense againstFDI attacks, where state-of-the-art methods adopt linear algebraand deep learning [39, 40]. However, works that address FDIattacks tend to make unrealistic assumptions on the adversary’sknowledge and capabilities.
2.6 Motivation for RESLab
RESLab aims to facilitate academic research and to bridge the-ory to practice through collaboration with industry and academia.Therefore, it must exemplify these qualities: (1) Ability to vali-date domain-specific use cases; (2) Ability to mimic and reflectthe real-world complexity of industrial systems by incorporatinghardware devices; (3) Support high-impact research activities;(4) Cost effectiveness; (5) Design for fast verification of results;(6) Ability to transfer results to power system industry; (7) Abil-ity to transfer the testbed itself to industry or other researchers;and (8) Ability to serve as an educational platform.
Because RESLab is a cyber-physical testbed, these features arealso important: (1) It is imperative that the testbed allows thedevelopment and evaluation of cross domain (e.g., cyber, physi-cal, protection system) analyses including identification of cyberthreats that impact power systems; (2) Ability to implement andvalidate realistic threats using real-time simulated models anddata as well as offline models; (3) Ability to evaluate cyber-physical contingency analysis involving simulating and patchingvulnerabilities and their impact on risk; (4) Ability to connectsimulators, emulators, and physical components.
These RESLab features fulfill the gaps in existing testbeds:
1. Virtualization and emulation: RESLab uses vSphereand CORE for virtualization, including power system
simulation in a dedicated VM, and to operate networkcomponents such as an IDS and firewalls.
2. Open source protocols and industry integration:RESLab’s implementation of OpenDNP3 follows theIEEE 1815 standard [41], hence convenient for otherresearchers to replicate the experiments. RESLab usesPWDS [29] to emulate the DNP3 outstations in real-time, making the solution easily deployable to industry.
3. Use case realism support: RESLab addresses use caseshortcomings by supporting multi-phase cyber intru-sions as presented by the joint report of NERC, E-ISAC,and SANS-ICS [42]. Other solutions fail to addressearly attack stages. Our testbed enables a complete rep-resentation of FDI and FCI attack vectors in a realisticenvironment from early-stage environment.
4. Large scale system cyber-physical analysis: RESLabenables research to develop entirely new systems. Forexample, it is supporting the use case of a cyber-physical EMS. Such a new system needs to supportalgorithms that enable cyber-physical state estimation.Its integration and application of large-scale realisticcyber-physical models, including a synthetic test caseon the Texas footprint that includes power [43] andcommunication [44] systems, with balancing authori-ties and market participants strengthens the test case tomimic a realistic cyber-physical power system.
5. Transferability and interoperability: The virtualizationin RESLab makes the migration to other platforms suchas VirtualBox and VMware simple. This approachmakes the testbed cost-effective compared to testbedsthat use RTDS or OpalRT.
6. Dataset management: Few datasets for cyber physicaltestbeds are publicly available. RESLab provides sucha platform that aggregates real-time traffic and powerdata along with IDS alerts and enables integration ofthird-party tools including visualization and data ana-lytics.
3 RESLab Cyber-Physical Testbed Architecture
RESLab is designed to reflect realistic power and cyber compo-nents based on the synthetic electric grid model on the Texasfootprint [43], where its communication model is introducedin [44]. Fig. 1 presents a high-level view of the RESLab archi-tecture, showing an example of one substation and one utilitycontrol center (UCC), with their power system cyber-physicalcomponents and data flows. More detailed data flows whichinclude balancing authorities and demilitarized zones (DMZs)are presented in our prior work on firewall policies that followNERC standards [46].
In the simplified model of Fig. 1, the main data flow depictedis DNP3 traffic, which is initiated at the UCC, where there isa DNP3 master and a SCADA server that act as our centralcontrol and human machine interface (HMI) applications. Atthe substation level, there is a DNP3 outstation (DNP3 O/S)which has the data from the field devices that the UCC needs tomonitor.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 4
Tabl
e1:
Rev
iew
ofC
yber
-Phy
sica
lPow
erSy
stem
Test
beds
Ref
Pow
erSi
mul
ator
sC
yber
Sim
ulat
ors
Com
mun
icat
ion
Prot
ocol
sD
evic
esan
dSo
ftw
ares
Syst
emL
evel
Intr
usio
nsD
ER
[23]
RT
DS
No
IEC
6185
0R
elay
s,IE
Ds,
Gat
eway
Subs
tatio
nN
oN
o
[24]
RT
DS
Net
wor
kSi
mul
ator
-3/
Det
erL
abPM
U/C
37.1
18PM
Us,
phas
orda
taco
ncen
trat
or(P
DC
),G
PScl
ock
Tran
smis
sion
DoS
,MiT
MN
o
[25]
RT
DS,
Opa
l-R
TW
ide
Are
aC
omm
unic
a-tio
nE
mul
ator
C37
.118
PMU
s,SD
N,R
TAC
,PD
C,I
ndus
try-
grad
eSC
AD
ATr
ansm
issi
onM
iTM
No
[7]
OPA
L-R
TR
ealN
etw
ork/
SDN
DN
P3SE
L35
1Tr
ansm
issi
onD
oSan
dco
ordi
nate
dph
ysic
alco
ntro
lN
o
[5,6
]R
TD
SR
ealN
etw
ork/
SDN
MO
DB
US/
TC
P,IE
EE
C37
.118
PMU
,PD
C,
rela
y,In
dust
ryso
ft-
war
efr
omSE
L,
GE
,Sno
rt,
Wir
e-sh
ark
Tran
smis
sion
Aur
ora,
DoS
No
[26]
RT
DS
DN
P3Sc
apy
Tran
smis
sion
MiT
MN
o
[32]
Typh
oon
HIL
602
No
IEC
6185
0D
igitl
Sign
alPr
oces
sor
(DSP
),FP
GA
,Sun
Spec
Syst
emV
alid
atio
nPl
atfo
rm,I
nver
tera
ndco
nver
ter
Dis
trib
utio
nN
oY
es
[33]
Pow
ersi
mD
ynam
icL
ink
Lib
rary
(DL
L)
Wifi
with
MO
DB
US
regi
ster
,SSH
IoT,
rela
yD
istr
ibut
ion
No
Yes
[34]
Pow
erW
orld
,M
AT-
LA
B,
RT-
LA
B,
OP5
600
Rel
ayD
istr
ibut
ion
No
Yes
[27]
RT
DS
No
PLC
,Rel
ays
Dis
trib
utio
nN
oY
es
[37]
Opa
l-R
Tan
dFP
GA
No
C37
.118
,Pr
ecis
ion
time
prot
ocol
(PT
P)D
istr
ibut
ion
No
No
[11]
RT
DS
Opn
etM
OD
BU
SL
ibM
odbu
s,O
PNE
T,R
TL
AB
Tran
smis
sion
(Atta
ckon
Stat
icV
arco
mpe
nsat
orco
ntro
ller)
MiT
MN
o
[8]
RT
DS
real
netw
ork
IEC
6185
0IT
AC
AID
Sto
olTr
ansm
issi
on(A
ttack
onSt
atic
Var
com
pens
ator
cont
rolle
r)32
type
sof
atta
cks,
MiT
MN
o
[35]
Pow
erW
orld
DS
Wid
ear
eaco
mm
unic
a-tio
nE
mul
atio
nD
NP3
and
GO
OSE
NI
CR
IO,
SEL
421,
SEL
651R
,SE
L73
4BTr
ansm
issi
onN
oN
o
[10]
RT
DS
SDN
base
dsw
itch,
fire-
wal
lIE
CC
37.1
18,I
EC
6185
0,D
NP3
Self
deve
lope
dSC
AN
VIL
LE
,R
AD
ICL
No
spec
ific
use
case
RA
DIC
Lfo
rcyb
erat
tack
sN
o
[9]
Rea
lhar
dwar
eR
ealh
ardw
are
No
spec
ific
use
case
RA
DIU
SD
oS,I
CT
Wor
min
-fe
ctio
n,M
ALW
AR
Ein
fect
ion,
Phis
hing
atta
ck,D
NS
pois
on-
ing
No
[13]
Pow
erW
orld
,M
AT-
LA
B,
RT-
LA
B,
OP5
600
OPN
ET
MO
DB
US-
RSI
mN
osp
ecifi
cus
eca
seD
oS(c
ompr
omis
edH
MI)
,SY
NA
CK
flood
ing
No
[45]
Pow
erW
orld
RIN
SE(n
etw
ork
emul
a-to
r)M
OD
BU
S-T
CP
Tran
smis
sion
DD
oSat
tack
No
[12]
OPA
L-R
TO
MN
ET
++
MO
DB
US
Am
etek
MX
-45
bi-d
irec
tiona
lgri
dsi
mul
ator
fora
mpl
ifica
tion
Mic
rogr
id(I
EE
E13
)N
oN
o
[14]
OPA
L-R
TN
osi
mul
ator
orem
ulat
orD
NP3
and
MO
DB
US,
C37
.118
Labv
iew
CR
IO,O
SIso
ft’sP
I-Se
rver
No
No
No
[36]
OPA
LR
Tan
dTy
-ph
oon
HIL
No
Xili
nxV
irte
x6
FPG
As
mic
rogr
id:
spec
ific
toge
nera
torc
ontr
olN
oN
o
RE
SLab
Pow
erW
orld
DS
CO
RE
DN
P3R
TAC
,Sn
ort,
Pack
etbe
at,
Ope
nDN
P3Tr
ansm
issi
onM
iTM
and
DoS
No
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 5
SCADA Server +
HMI
DNP3 O/S
Switch
DNP3 Master
Router/Firewall
Rout
er/F
irew
all
RTU RTACs
Relays Relays
Analog Devices
Digital Devices
DNP3Substation
Utility Control Center (UCC)
Figure 1: Power system cyber-physical architecture with onesubstation and a utility control center.
The RTUs, RTACs, and relays monitor the system status, collectdata, and control physical devices such as circuit breakers, whichwe call digital devices as they have only two states, or generatorsand load which we call analog devices. The RTUs and RTACscan control generators’ output and affect loads. The relayscan trip a circuit breaker to isolate a faulted circuit. The datafrom the Process Level is concentrated in RTUs and RTACs andthen transferred to the Substation Level. Within the UCC, theDNP3 master collects the information from each substation fora complete view to understand and control the system. In [47],a typical structure of data concentration and engineer access ispresented using the SEL RTAC in SCADA systems, where theRTACs in substations communicate with RTACs in UCC andEMS for data collection and control.
The UCC and substations are in different locations, and inRESLab they are interconnected by an IP network, but theycan also be connected through a serial link. At each locationwe have one router: the substation router and the UCC router,which also act as firewalls because they are configured to allowonly DNP3 packets and block unwanted traffic. Fig. 2 illustrateshow RESLab follows this data flow pipeline and incorporatesreal-time power system simulation using PWDS, a physicalSEL RTAC, an OpenDNP3 master application, and an emulatedcommunication network using CORE.
In the testbed, PWDS acts as a collection of DNP3 outstationsconnected to the substation’s control network (shown as SubLAN in Fig. 2). The emulated DNP3 master and SEL RTACare housed in a control center network to represent software-and hardware-based control platforms. Each of the emulatedcomponents are hosted in a virtual machine management en-vironment, named vSphere. The vSphere environment allowsfor the creation and management of a large number of VMs. InRESLab, connections between emulated and physical compo-nents are made to scale the network depending on the use case.Next, the purpose and functionality of each testbed componentis analyzed.
Figure 2: The logical connections between the VMs hosted inRESLab.
3.1 Cyber network emulation: CORE
RESLab uses CORE, which is an open source network emulatorpublished by the U.S. Naval Research Laboratory. The soft-ware allows the creation of several BSD jails, which are similarto Linux containers, that can be connected to emulate realisticcommunication networks. These containers are used to emulaterouters, firewalls, personal computers, and Linux servers in thecommunication network. CORE can also tap into the hosts’ Eth-ernet connections to connect with external networking devicesand VMs housed within vSphere.
In our testbed, CORE is hosted as one of the VMs, with eachof its virtual network interfaces connected to different VLANs,such the Sub LAN and Utility LAN shown in Fig. 2, to emu-late a wide-area-network (WAN) between substations and UCC.CORE also has a bridge connecting the cyber-physical EMSapplication (Section 3.5) that monitors real-time traffic fromPWDS as well as network traffic in CORE. The WAN setuphas direct connections between the gateway routers of the UCCand substation subnets. The routes within this architecture arecreated by running Quagga [48] services in the routers whichemploy open shortest path first (OSPF) [49] as the routing pro-tocol.
From left to right in Fig. 2, the connections are: (1) VM host-ing DNP3 master, (2) VM running CORE, (3) VM running acentralized cyber-physical EMS application, and (4) the PWDSVM. To show the emulated network, Fig. 3 details the networktopology: the DNP3 master and SEL-RTAC are connected to theCORE through virtual interface [1]; interface [2] forwards SnortIDS alerts from control center router to the EMS application;the VM running the large-scale synthetic electric case in PWDSis connected through interface [3].
3.2 Power system simulation: PWDS
In RESLab, PWDS models the dynamic behavior of an electricpower system in the transient stability time frame. It does inter-active control [43], and serves as a general interface for DNP3outstations [29, 30]. An outstation, as defined from a powersystem operational point of view, typically includes one substa-tion and its devices, including branch breakers, generators, loadbreakers, and shunts. The DNP3 tags generate binary data, suchas the status of all devices, and allow the devices to be controlledby other DNP3 masters/clients. The DNP3 tags can also be setto send analog data, such as measurements of generator real andreactive output, branch power flow, and bus voltage, as well as
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 6
Figure 3: CORE network topology showing emulated PC nodesand connections to: [1] DNP3 master, [2] CYPRES app, and [3]PWDS DNP3 outstations.
allowing DNP3 masters to change generator setpoints. A DNP3master is hosted in a VM running OpenDNP3. Another VM isrunning SEL acSELerator software [50] and is used to configurethe RTAC as a DNP3 master.
PWDS also serves as a simulation engine with a generic interfacefor integration into other applications [31]. In our experiments ,PWDS simulates the power system in a real-time environment inwhich cyber threats and defense mechanisms are implemented.The large-scale test case on the Texas footprint [43, 44] is imple-mented as our exemplar power system and is being maintainedat Texas A&M. CORE’s WAN is being used to forward breakerstatus and control commands between VMs.
3.3 DNP3 and master application
DNP3 is extensively used by electric utility companies in NorthAmerica for communication between equipment [51]. The pro-tocol utilizes the master/outstation architecture. A network canbe configured to have one DNP3 master communicate with morethan one DNP3 outstation, referred to as a multi-drop network.Alternatively, there can be one DNP3 master that communi-cates with one DNP3 outstation, or a one-on-one network, asimplemented in this paper.
DNP3 messages contain a 10-octet DNP3 header and a maxi-mum 292-octet DNP3 payload, which are carried over TCP/IPpackets. The DNP3 header contains sync, length, link control,destination, and source address fields with a cyclic redundancycheck (CRC) to ensure data integrity. The DNP3 payload iscomprised of many 16-octet data blocks, with a 2-octet CRC foreach block.
The purpose of the CRC is to ensure that bits have not beenchanged accidentally during its journey from source to endnode. Some intruders may modify the traffic yet fail to modifythe CRC which can be easily detected at the receiver or byimplementing DNP3 specific decoders in IDS. Inside the DNP3payload, function codes identify the operation the outstationperforms. The index identifies the device in the outstation themaster is asking to perform the operation on or retrieve data from.These are the function codes used in our simulations: Confirm(0x00), Read (0x01), Read (0x2), Select (0x03), Operate (0x04)Direct Operate with Acknowledge (0x05), Solicited Response(0x81), and Unsolicited Response (0x82).
The DNP3 master application in RESLab uses the PyDNP3 li-brary, a Python wrapper for the C++ based OpenDNP3 module,to run the master as a console and a graphical user interface(GUI) application. The purpose of the master application isto continuously monitor the status of the circuit breakers, gen-erators, and loads in the DNP3 outstations that are running inPWDS. The application also forwards the response of DNP3outstations, as well as connection status, to the central appli-cation via CORE’s WAN. This application is configurable tochange the polling rates and visualize real-time traffic. It runsin an isolated VM but exists in the UCC LAN with its defaultgateway set to 172.16.0.4, which is the UCC router (see Fig. 3).
3.4 RTAC integration
RESLab also incorporates the SEL-3530 RTAC to explore dif-ferent variants of DNP3 master. The RTAC provides flexiblesystem control with integrated management of security, configu-ration, and logic. It supports multiple communication protocols,such as DNP3, Modbus, and IEC 61850, and comes with anembedded IEC 61131 logic engine [52]. RTAC has been utilizedin several hardware-in-the-loop testbeds for data collection andsignal conversion [25, 35], but they do not use it for communi-cation studies or to emulate cyber adversaries associated withspecific hardware.
Within RESLab, for each substation there is a DNP3 masterin the RTAC to collect analog input data, such as power flow,current, and generator output, in addition to binary input data,such as the status of branches, generators, loads, and shunts fromPWDS. Furthermore, each client in the RTAC can control thecorresponding devices through analog and binary outputs suchas to change generator setpoint in Mega Watts (MW) and devicestatus (on/off). Thus, the integration of an industrial standardcontrol device in RESLab allows researchers to gain a deeperunderstanding of how cyber adversaries can impact the devicesand the system as well as develop more practical detection anddefense logic in the field.
3.5 Cyber-physical energy management system
A centralized cyber-physical energy management applicationthat our team developed named Cyber-Physical Resilient EnergySystems (CYPRES) is designed to house algorithms for mon-itoring and analysis, run SCADA applications, and visualizethe system. CYPRES is developed and deployed in RESLabas an exemplar use case for the testbed. CYPRES aggregatesinformation from the cyber side CORE emulation environment,the power side from PWDS, as well as from the DNP3 mastersregarding DNP3 communication status. CYPRES is used tovisualize the control network of the synthetic utilities and theirsubstations in the synthetic power grid. To detect instrusions,it also probes real-time traffic, where CYPRES then performsdata fusion from multiple sensors in the synthetic network. TheCYPRES application is currently envisioned to be housed at acentral location (i.e., at a balancing authority or utility) and usedto analyze the system with respect to cyber intrusions. Further-more, CYPRES provides cyber-physical situational awarenessin RESLab using attack tree visualizations. These visualizationscan be tailored to network administrators or to power systemoperators to provide an actionable map of risk related to cyberand physical assets and impact, and to recommend mitigation ac-
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 7
tions for the identified risks within a network, e.g., by informinghow to protect against cascading failures.
3.6 Intrusion detection system
The role of an IDS is to detect cyber intrusions. Rule-basedand anomaly-based IDS’s are predominantly used in industry,but they lack the capability of detecting zero-day attacks. Asan initial approach, RESLab integrates the Snort IDS which isused to detect and generate alerts for cyber intrusions. Snorttutorials [53] are followed to define rule sets, preprocessors,decoders and change configuration. Currently in RESLab, Snortis protecting the control center and substation LANs by runningas a service within the routers. The alerts are forwarded to theCYPRES application in real-time.
For the synthetic power system case, the dataset from the attackscomprises packets with a destination port of 20,000, which is thedefault DNP3 port. From the filtered dataset, the frequency ofcommunication between the master and outstation is analyzedin Section 6 to show to the effectiveness of the exemplar cyberintrusion scenario in impeding DNP3 communications.
3.7 Storage and visualization
RESLab implements a platform that the team has created toprobe the traffic at all the network interfaces inside CORE, tocollect the traffic, to use Elasticsearch Logstash Kibana (ELK)stack to store the traffic in an Elasticsearch index, and to vi-sualize them using Kibana dashboard with the Packetbeat plu-gin [54]. One can configure the Packetbeat plugin to modify thenumber of interfaces and the type of traffic to probe. Kibanaprovides a platform to write Lucene queries to filter out a searchin the Elasticsearch index. RESLab uses Logstash to collectSnort alerts to visualize in Kibana. In addition to ELK stack,RESLab also integrates Zabbix [55] for network monitoring, asit provides a platform to configure custom alert rules and trig-gers. We have configured a Zabbix server in the base operatingsystem hosting CORE, and the Zabbix agents in all the routers inCORE. The agents within CORE use the CORE control networkto interact with the server using ZBX protocol [55].
4 ThreatModel
Widely-known cyber threats such as the Ukranian and Stuxnetattacks have been multi-stage attacks, which are a serious con-cern. However, due to the nature of these attacks, and the widerange of time scales involved at each stage, they are challengingto plan and study. Hence, RESLab is intended to mimic anelectrical utility environment allowing for experimentation ofindividual threats, which enables us to develop and test solutionsat each step in the attack vector.
The threat model we present and implement in this paper is basedon emulating a multi-stage attack in the large-scale synthetic testsystem’s communication model. In the first stage, the adversarygains Secure Shell (SSH) access to a machine in the substationLAN. In the second stage, the adversary performs steps that aretailored to the system under study and to power system protocols,allowing the adversary to achieve MiTM and DoS attacks that
cause physical impact. The RESLab framework can not onlysupport MiTM and DOS, but can also integrate other attackvectors.
4.1 Man-in-The-Middle (MiTM) attack
MiTM is one of the oldest forms of cyber intrusion, where aperpetrator positions itself in a conversation between two endpoints, to either passively eavesdrop or to impersonate one of theendpoints, making it appear to be a normal exchange of infor-mation. MiTM encompasses different techniques and potentialoutcomes, depending on the threat model. During the secondstage of our presented threat model, we compromise the targetoutstation and its router through performing an ARP spoof attackby poisoning the ARP cache of both the substation’s gatewayand DNP3 outstation [56]. Then, in the third stage, we modifycontrol and monitoring traffic to have different implications onthe electric grid.
Such tampering of commands and measurements would nor-mally go undetected by the outstation using CRC error checking,since the data chunk in the DNP3 payload’s has its CRC recalcu-lated by the adversary before the modified packet is forwardedto the outstation. The intruder causes false command injection(FCI) and false data injection (FDI) attacks by first storing theDNP3 polling response for the targeted outstations, then manip-ulating measurements in some cases, and commands in othercases, as well as manipulating a mix of both to carry out one ofthe most critical contingencies presented in our N-x contingencydiscovery paper [57]. Such an attack is hard to be detected by anIDS such as Snort, if the intruder not only tampers the commandand takes care of the CRCs.
In RESLab, the MiTM attack is developed and implemented tochange binary and analog commands sent by the DNP3 masterto the outstation as well as the polled response from DNP3outstations. The intruder not only modifies commands but alsoeavesdrops and then modifies the current state of the system bytampering its real-time measurements. In Table 2, the procedurefor performing a MiTM attack in RESLab is listed. The detailson the various combination of attacks that are performed in thethird stage of the threat model is presented through four usecases detailed in Section 5.
Seq. Description1 Start the CORE, PWDS, OpenDNP3 master.Allow
time for DNP3 communication between master andoutstation to be established.
2 Start CYPRES app. to monitor cyber data.Start run-ning Snort in substation router. Run the ELK ser-vices and Packetbeat .
3 ARP cache poisoning of substation’s gateway andoutstation.
4 Sniff traffic to and from the outstation.Forward nonDNP3 traffic to/from outstation.
5 Send command from master to outstation. Modifycommand and forward to outstation.
6 Modify TCP acknowledgement (ACK) from outsta-tion.
Table 2: The steps taken in RESLab to implement FCI injection.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 8
4.2 Denial of Service (DoS) attack
As another attack vector, we implement a DoS attack to exhaustvictim nodes’ processing capability and link bandwidth. Thereare many different methods of DoS attacks that can be used,which include but are not limited to: user datagram protocol(UDP) flood, internet control message protocol (ICMP) flood,and Ping of Death (PoD) [58]. While each of these DoS attacktypes use different Open Systems Interconnection (OSI) layerssuch as application, presentation, session, transport, network,data link or even at physical layer protocols to carry out theattack, all DoS methods attempt to disrupt the communicationchannels of the targeted node. In our threat model, the intruderwithin the substation LAN targets routers at the substation and atthe control center by flooding the routers with ICMP traffic. Theimpact of these DoS attacks is then observed and analyzed basedon round trip times (RTT) and throughput of the communicationchannel by varying the strength of the attacks such as the lengthand delay between the ICMP packets infused to disrupt theDNP3 session.
5 Impact of Cyber Threats on Power OperationThe synthetic Texas 2000-bus case [43, 44] is a publicly avail-able power system test case. This system is N-1 secure, whichmeans the system can still operate securely with one deviceis out, and it is difficult to cause disruption by exploring N-2contingencies. Hence, the use cases in RESLab leverage re-sults from our prior work [57] on identifying the most criticalmultiple-element contingencies based on graph theory and lineoutage distribution factors (LODFs), which are located in theregions targeted in our use cases (Fig. 4).
Assume branch (x,y) means from Bus x to Bus y in the Texas2000-bus model. To illustrate the contingencies, if branches(5262,5260), (5263,5260), (5317,5260), (5358,5179) are open,there will be four overflow branches in the system, which arebranches (5071,5359), (5138,5071), (8086,8083), (8084,8083).Branch (5262,5260) and (5263,5260) are located in SubstationGLEN ROSE 1 (560), Branch (5317, 5260) is at SubstationGRANBURY 1 (601), and Branch (5358, 5260) is at Substa-tion RIESEL 1 (631). The overflow branches are at SubstationWACO3 (399), JEWETT1 (1195) and FRANKLIN (1200).
Besides, in those substations there are several generators. Wehave studied that if those generators are compromised, therewill also be a contingency in the system. These generators areGen 5262, 5263, 5319, 5321, 5360, 7098, 7099. Gen 5262and 5263 are at Substation GLEN ROSE 1 (399), Gen 5319and 5321 are at Substation GRANBURY 1 (601), Gen 6360is at Substation RIESEL 1 (631), and Gen 7098 and 7099 areat Substation WADSWORTH (968). When these generatorsreduce their output and the Branch (5260,5045) in SubstationSTEPHENVILLE (390) is open, there will be another overflowin Branch (5286, 5046) at Substation STEPHENVILLE (390).
Thus we assume that the adversary has the intent and the re-sources to target the most critical branches and generators,where disrupting their control causes severe impact. Specifi-cally, we present four use cases to show how cyber threats cancompromise a resilient power system. These use cases involvebinary and analog command modification, measurement andstatus modification. Before exploring the scenarios, we present
Figure 4: The Texas-2000 bus model is the basis of our exemplarcyber-physical power system.
RESLab’s experimental setup which allows us to collect dataat various locations and analyze them from a cyber-physicalperspective.
5.1 Experimental setup
The DoS and the MiTM attacks for the scenarios are performedwhile running RTAC and OpenDNP3 applications as the DNP3masters. The resources used to perform all the experimentsare illustrated in Table 3. Virtual LANs (VLANs) are used toensure that traffic is forwarded by the emulated routers in COREand to segregate the substation network from the control centernetwork.
In these simulations, we use a multi-master architecture whereeach master monitors and controls a substation separately. Whilethe master monitors and controls outstations, the adversary sniffsall the measurement traffic (requests and responses) from thesubstations. We capture network traffic at four locations in thenetwork (outstation, master, adversary and substation router) toevaluate the impact of MiTM attacks on these four use cases.Since the adversary acts as the middle man between the substa-tion router and outstation, we validate the MiTM by checkingif the DNP3 packets received at the substation router and at themaster are identical, and if the DNP3 packets at the outsationand adversary are identical. To test the detection of DoS andMiTM attacks, we operate Snort at the substation and controlcenter routers in a Network Intrusion Detection System (NIDS)mode by enabling preprocessors and decoders and including cus-
Virtual machine allocations in vSphereVM Name Mem. CPU
PWDS 10G 2 1,2,4 Windows 10Central_App 16G 8 1,2 Windows 10RTAC 4G 2 1,2,3 Windows 10
Table 3: VM configuration for the RESLab architecture in Fig. 2.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 9
Figure 5: UC1: Overloaded transmission lines observed atmaster application (WACO 3 (399), WACO 1 (456), JEWETT1 (1195), and FRANKLIN (1200)). The legend shows theoutstation_index, for example, the first legend indicates out-station number 399 and DNP3 index 5. The plot beneath showsthe Snort alerts during the intrusion.
tom rules for ARP, DNP3, and ICMP traffic. Then, we presentthe alerts along with the physical traffic to correlate the alertswith the measurements and command tampering.
5.2 Class 1: False Command Injection (FCI)
A MiTM attack that modifies binary control commands usingrelay control blocks can cause line overloading [59]. To achievethis, the adversary first parses the measurements by sniffing theDNP3 responses from the outstation. Then, it sniffs the DNP3binary OPERATE command and forges them. The adversarymodifies commands with function codes of 3 and 4 (SELECTand OPERATE command) from the RTAC, and it modifies com-mands with function code 5 (DIRECT OPERATE command)from the OpenDNP3 master application. The adversary mod-ifies all the CLOSE commands to TRIP, forcing to open thecritical branches identified and causing line overloads in fourother branches, shown in the data from the scenario in Fig. 5.This scenario is referred as use case 1 (UC1).
The intruder can also modify analog control commands tochange the setpoints in generators along with a binary com-mand to control a branch to cause line overloads. The intruderfirst inspects the DNP3 packets, changes a collection of gen-erator setpoints from the real value to 0, and alters the binarycontrol command as in UC1. This scenario compromises sevengenerators and one branch, referred as use case 2 (UC2).
Fig. 6 shows the actual generation output in each substationWADSWORTH, RIESEL, GRANBURY, and GLEN ROSE alongwith the Snort alerts during 5th, 10th, and 11th mins of the sce-nario. The intrusion in these substations takes place during the8th and 9th mins. The intruder’s goal is to overload the trans-mission line near substation STEPHENVILLE, accomplished bytampering of the analog set points, as observed in the interval9-11th minute in Fig. 6.
Figure 6: UC2: The real power injection at generators fromsubstations WADSWORTH (968), RIESEL 1 (631), GRANBURY1 (601), GLEN ROSE 1 (560) (left y-axis) and the overloadedline near substation STEPHENVILLE (390) (right y-axis). Thelegend shows the outstation_index. The plot beneath shows theSnort alerts.
5.3 Class 2: False Data Injection (FDI) with FCI
The MiTM intruder can also perform FDI with the FCI to cre-ate more difficult-to-detect attacks. First, the intruder falsifiespolled measurements, causing the operator to re-send a controlcommand to the field device. Then, the intruder modifies thecontrol command, as in the previous use cases, by changingthe generator setpoint. The actual generation measurementsfor the same seven generators in UC2 are falsified to 20 MW,and the flow measurement coming from branch [5260, 5045]is changed to 3000 MW, which is above its capacity. Based onthese observations, the operators or a pre-defined control logicwithin devices such as an SEL RTAC, would re-send the controlcommand to increase the generators’ output and open the branch.However, once sending those commands, the intruder modifiesthe setpoints to 20 MW, making the physical system unreliable.This scenario is referred as use case (UC3).
Fig. 7 shows the system after the output of a generator in sub-station WADSWORTH is changed in the polled measurementsby the intruder from 1000 MW to 20 MW as observed in themaster and the router during 52nd to 55th min. The Snort alertsare observed from the 53rd to 56th mins. The alerts at 50th and51st min are due to an attack in other targeted substations suchas GLEN ROSE, RIESEL, GRANBURY, whose generation setpoints are tampered.
Another example of a three-stage attack is referred as use case 4(UC4), where the intruder first changes the measurements polledby the DNP3 master, as in UC3. Once the operator re-sendsthe control command, the intruder changes the setpoints fromthe real value to a low value, as in UC2, but the intruder alsofalsifies the measurement packets, masking the true measure-ments but showing the original setpoint values. The result is thatthe operator believes his/her command has been successfullyreceived and committed. However, in the true physical system,the generators’ outputs are decreasing, and opening a line willthen cause an overload.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 10
Figure 7: UC3: The real power injection at one generator in sub-station WADSWORTH as observed by master (mas), substationrouter (rou), adversary (adv), and the outstation (os), along withthe overloaded line (of_ln) near substation STEPHENVILLE.The plot beneath shows the Snort alerts during the intrusion.
Figure 8: UC4: The real power injection at one generator in sub-station WADSWORTH as observed by master(mas), router(rou),adversary(adv), and the outstation(os). The overloaded linemagnitude (right y-axis) near STEPHENVILLE (of_ln).
Fig. 8 shows the generation output at substation WADSWORTHas observed at four locations. During the intrusion onWADSWORTH, within the 34th and 44th mins, the adversary firstforces the master to take a wrong action to change the generationoutput to 1000 MW once the master observes low generationoutput at the 34th minute due to modification of the measure-ments of generation output. Further, when the operator takes thisaction to address the low generation output, the intruder changesthe command from 1000 MW to 0 MW to cause contingency. Tobe stealthier, the intruder also modifies the polled response fromoutstation with the same setpoint value of 1000 MW from theinterval 39 to 44 min except at 42nd min, as set by the operator,to prevent the master from observing the contingency caused bythe intruder in first two stages. The snort alerts generated in thisinterval are shown in Fig. 8.
6 Results and AnalysisIn this section, DoS and MiTM attacks are performed and ana-lyzed on the DNP3 sessions between the masters and outstationsbased on the four use cases, which are summarized in Table 4.The effectiveness of the DoS attack is evaluated by varying at-tack strength and studying its impact on the RTT and throughputof DNP3 traffic. The time frame of power system operationscompared with the attack time frame plays a major role; for ex-ample, the time frame of inverter and stator transient control arein the order of milliseconds, while control of voltage stability,power flow, and unit dispatch range from 10 to 1000 seconds.Hence, it is essential to minimize RTT to ensure that the controlcommands are processed by the field devices on time.
The use cases for MiTM attacks are tested with the RTAC andOpenDNP3 master. Experiments are conducted by varying thenumber of DNP3 masters, as well as the polling interval. Eachmaster communicates with its substation, and we assume that5 or 10 master are connected with their respective outstations.These experiments are performed to study the success rates ofthe attacker in causing the desired contingency of each use case.The adversary is restricted by the available resources in the Linuxcontainers in CORE, thus the attacks are stochastic in nature. Asthe number of masters increases, the amount of traffic an intruderprocesses increases, which results in higher attack miss rates,i.e., the probability that the attacker fails to modify a sniffedpacket. The results demonstrate the effective implementationof the use cases by observing the real-time physical side data atdifferent locations in the testbed.
We also study the number of active TCP connections as impactedby retransmission during the progression of the attack based ondifferent polling rates and varying DNP3 masters. The adversarysuccess probability, the average retransmission rates, the packetprocessing times, and the average RTT for performing each FCIand FDI attack, and the Snort alert statistics are key characteris-tics for detection. Snort IDS is used to detect the ICMP floodattack as well as the ARP spoof attack (Section 4) that reroutespackets to the adversary and allows modifications to take place.
6.1 DoS attack evaluation
The DoS attack is performed by increasing the ICMP payloadsize for a fixed interval rate, as well as by varying the intervalrate with the fixed payload size, to determine their impact on theRTT and throughput of DNP3 traffic. For all the experiments,the virtual Ethernet communication links in the network have afixed bandwidth of 10 Mbps, with a 160 µs transmission delay.
We first run polling at an interval of 30 and 60 s and DIRECTOPERATE commands from the DNP3 master without any at-tack, to verify the network performance. Then, either the delayinterval or payload length of DoS traffic is altered, and the samepolling and control operations are performed.
The DoS attack is performed by the compromised device inthe substation, encircled in red, as shown in Fig. 3. The DoSattack is directed at two targets: the substation router and thecontrol center router, and we seek to determine which device isimpacted more by a DoS event. From the DoS trials directed atthe substation router, we observe that only the broadcast domainof the substation LAN is exhausted. However, if the attackis directed at the control center router, the link between the
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 11
Figure 9: Impact of DoS on RTT by varying payload size.
Figure 10: Impact of DoS on RTT by varying attack interval.
two routers as well as the broadcast domain of the substationLAN are exhausted. Both DoS attacks are performed usinghping3 [60] while keeping a fixed interval rate of 1000 ms andincreasing the payload size of the ICMP packets from 800 bytesto 1800 bytes in increments of 200 bytes for each trial. It can beobserved from Fig. 9 that average RTT increases with payloadsize, and that the attack on the control center router has a higherimpact in comparison to substation router.
Further, DoS attacks are performed by keeping the payloadsize of the ICMP packet fixed at 1000 bytes while decreasingthe ICMP packet arrival rate from 1500 ms to 500 ms in stepdecrements of 100 ms for each consecutive trial. Fig. 10 showsthat the average RTT decreases with increase in attack intervalas well as the lower attack interval has higher impact on thecontrol center router in comparison to the substation router.
A DoS attack primarily affects the target’s downstream band-width. Hence, the average throughput will be affected as thebandwidth of the link is affected. The average throughput forthe substation router is calculated using the transmission time ofDNP3 packets, as per Eq. 1:
Average Throughput “Total data payload in bytes
Total transmission time(1)
The average throughput depends on the command type fromthe DNP3 master. For example, the response payload size forthe polling will be quite high compared to the response of the
Figure 11: Impact of DoS on varying payload size on averagethroughput and goodput at both substation and control centerrouters.
Figure 12: Impact of DoS on varying attack interval on averagethroughput and goodput at both substation and control centerrouters.
OPERATE commands. The goodput is equal to the throughputif there are no retransmissions.
In Fig. 11, we observe that the throughput and goodput increaseas the payload size increases up to certain extent, then they de-crease due to congestion in the network. It can also be observedthat the difference between throughput and goodput increasesas the payload size increases due to high retransmission causedby the congestion. Similarly, reduced goodput is also observedwhen the attack interval is lowered from 1500 ms to 500 ms asseen in Fig. 12.
6.2 MiTM attack evaluation
In the MiTM attacks, both master and outstation DNP3 packetsare captured at the adversary’s machine located in substationLAN. Fig. 13 shows Wireshark sniffing the DNP3 DIRECT OP-ERATE command from the master in addition to the responsefrom outstation. As described in Section 5, the CLOSE com-mand is replaced by the TRIP command as observed from theresponse, as well as the DNP3 log of PWDS as seen in Fig. 13.
The RTT for MiTM attacks is small compared to the RTT for aDoS attack. In a DoS attack, the RTT depends on the number ofretransmissions, but in a MiTM attack, the RTT depends on howmuch time the attacker takes to parse the packet, modify the
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 12
Figure 13: The DNP3 DIRECT OPERATE command altered bythe intruder.
Figure 14: Verification that intruder used the same sequencenumber to forward modified packet.
payload, recalculate the checksum and CRCs, and to forwardthe packet to the target. There is no substantial retransmissionin the case of MiTM attacks, if the intrusion is stealthy.
The occurrence of a MiTM attack is validated both by observinga rise in RTT compared to the normal operation in Fig. 15 andfrom its sequence number graph Fig. 14. Specifically, in Fig. 15,the MiTM attack is performed from 200 s to 1000 s, and the RTTis observed to increase to almost 150 ms during sniffing and FCIattack and to almost 200 ms during FDI attacks on measurement,indicating the time taken by the adversary’s machine for parsingand modification affects the overall RTT. Additionally, as thesequence number remains at 18 from 3.3 s to 3.4 s in Fig. 14, itindicates the attacker used the same sequence number to forwardthe modified packet.
6.3 Use case specific physical impact evaluation
The physical impact is evaluated based on the four use casesshown in Table 4, described in detail in Section 5. The targetof the MiTM intruders in UC2, UC3, and UC4 is the same butthey adopt different strategies to accomplish it. These use casesdetailed in Section 5 demonstrate increasing complexity. Thetime to cause the same overload in branch [5286, 5046] differsbased on the strategy in each use case, as illustrated in Fig. 16.For Use Cases 2, 3, and 4, the overload occurs at 173 s, 216s and 541 s, respectively. The differences in time as well asthe system dynamics are due to the amount and sequence ofintrusions in these three strategies.
Figure 15: RTT of DNP3 traffic through the Substation Routerduring the FCI and FDI attack in use case 4 at substationWADSWORTH.
6.4 Evaluation of MiTM attack practicality
The successful implementation of the attack use cases requiresthe intruder to cause the binary operate (BO) and analog operate(AO) FCIs and read response (RR) FDIs in a particular sequenceas shown in Fig 17. Due to the resource limitations at theattacker, such as sniffing from a single network buffer, it canonly accomplish the modification operations with a successprobability of p, q, and r for BO, AO, and RR packets separately.Assuming the number of BO, AO and RR operations to be m, nand o, the expected number of steps that the intruder has to taketo reach its goal is inverse of the success probabilities, which ispmqn for UC2, ro pm for UC3, and pmr2o for UC4.
The intruder continues the attack until it reaches its goal to over-load the branch [5285,5046]. Hence, we evaluate the averageminimum number of FCI and FDI modifications the intruder hasto perform to reach its target. The success probability dependson the available resources of the intruder, the master polling rate,as well as the number of masters polling the measurements.
For UC4, the number of FDI attempts is higher because the pro-cessing time of an FDI is higher than for an FCI, as it involvesparsing the DNP3 response from outstations that usually havelarge payloads. This higher processing time reduces the successprobability of the FDI attack. For UC3, in the RTAC case, anexception of higher FCIs are observed due to the automatedgeneration protection control logics incorporated in the RTAC.The protection logic ensured that the generation setpoint in-creases when the generation output reading goes below a certainthreshold.
Table 5 shows the minimum number of FCI and FDI attemptson average that are required to accomplish the final goal ofthe intruder for each use case with both RTAC and OpenDNP3master.
From queueing theory, we know that the traffic intensity ρ iscomputed based on the packet arrival rate λ and the service rateµ as ρ “ λ
µ[61]. From intruder reference, the arrival rate λ is
determined by the polling rates from the master as well as thenumber of DNP3 masters. The service rate µ is fixed since it isthe single intrusion node that processes the incoming traffic. Thehigher the ρ, the lower is the success probability for the intruderto modify the traffic. In literature, the arrival rate distribution canbe deterministic or random. In our simulation, since we observepolled traffic as well as commands, it can be considered a random
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 13
Figure 16: Impact of line overload caused through different use cases.
Table 5: Minimum number of FCI and FDI attempts required onaverage by the intruder for accomplishing its goal in UC2, UC3,and UC4.
distribution. Every payload that the intruder fails to forwardresults in drop of the packet, and that triggers retransmissionsfrom the sender. The algorithms developed and utilized for FDIand FCI attacks affect the processing time.
6.5 ELK stack visualization
RESLab visualizes the results using ELK stack, where Fig. 18shows a real-time count of the number of active TCP flows whilethe experiments are being performed for the four use cases with5 and 10 DNP3 masters. Since the number of active TCP flowsis an indicator of the number of connected clients, it helps usto detect if there are more than intended number of clients. At
certain times, we observe more than 10 active connections, assome clients lose connection and re-initate a new connectionwith a different source port number due to the MiTM attacks.The number of active TCP flows are the indicator of number ofmasters connected. Higher variance of connections are observedin 10 master cases due to higher retransmissions.
The Kibana Query Language (KQL) filters helps us to filter traf-fic, based on the source IP of the DNP3 master (i.e., 172.16.0.2)and the destination port in the DNP3 outstations (i.e., 20000)as shown in Fig. 18. A separate Logstash index is created inElasticsearch to store real-time snort alerts. Fig. 19 shows thehistogram created with Kibana for different types of snort alerts(ICMP flood, ARP spoof, DNP3 operate) during one of thescenarios from the use cases.
Figure 18: Count of TCP flows from Packetbeat using Kibanawhile the use cases with 10 and 5 masters are incorporated.
6.6 Discussion
These results validate the integration of emulators, simulators,hardware, and software tools including visualization and IDS inRESLab by performing DoS and MiTM attacks on the power
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 14
Figure 19: Number of Snort alerts by alert type using Logstashand Kibana.
system. Through four use cases, RESLab shows how suchattacks can impact the electric grid.
To understand the dynamics of the DoS attack, results present theimpact on RTT and throughput due to different attack intervalsand payload sizes of ICMP injections in DoS. For understand-ing the dynamics of the MiTM attack, we analyze strategiesadopted by the intruder to cause the desired contingencies. Wealso explore the MiTM attacks with different polling intervalsand number of master applications, which impacts on retrans-mission rates, RTT, and packet processing time. The intrusionsperformed in UC3 and UC4 provide the platform to create andmitigate FDI attacks on state estimation which involves an in-truder tampering the measurements.
The simulations performed for the substation network in COREconsisted of one broadcast domain. This causes the intruder toobserve the traffic related to all the substations. The number ofDNP3 masters are limited to 5 or 10 in our scenarios which isenough to enable the intruder to accomplish its N´ x contingen-cies such that its x components are in these 5 or 10 substations.However, they are modeled through a single substation networkin CORE. Hence, the intruder’s capacity to inject modified traf-fic is resource-limited due to having a single substation LANin CORE, as the intruder can only process traffic on the singlenetwork buffer.
7 Conclusion
A cyber-physical testbed provides a platform to understand se-curity threat events and their impact on the power grid. Thiswill help to facilitate grid resiliency to cyber intrusions. Inthis work, we present our testbed RESLab, where its architec-ture makes use of components such as vSphere, CORE, PWDS,Snort, among others, to successfully emulate the physical andcyber component of a synthetic large scale electric grid, and wedemonstrate the use of DNP3 based control and measurementtraffic to and from substation field devices. The methodologyand mechanics behind our testbed are demonstrated throughfour use cases utilizing two types of cyber intrusion experiments:DoS and MiTM. The dynamics of the intrusions are validatedby implementing use cases targeting specific parts of the grid.These intrusion events are evaluated from their respective char-acteristic features, including latency (RTT), throughput, andgoodput in the emulated WAN network.
By providing a safe proving ground for cyber-attack experi-mentation, RESLab is a platform to study defense mechanisms,where its ability to generate real-time datasets and customizemonitoring, visualization, and detection will play a major rolein developing cyber-physical state estimation, situational aware-ness, optimal response, etc. to prevent impending contingencies.
8 Acknowledgements
This research is supported by the US Department of Energy’s(DoE) Cybersecurity for Energy Delivery Systems program un-der award DE-OE0000895.
References
[1] E. Targett. High Voltage Attack: EU’s PowerGrid Organisation Hit by Hackers, March 2020.URL https://www.cbronline.com/news/eu-power-grid-organisation-hacked.
[2] D Alert. Analysis of the cyber attack on the ukrainianpower grid, 2016.
[3] Lev Streltsov. The system of cybersecurity in ukraine:principles, actors, challenges, accomplishments. EuropeanJournal for Security Research, 2(2):147–184, 2017.
[4] Ralph Langner. Stuxnet: Dissecting a cyberwarfareweapon. IEEE Security & Privacy, 9(3):49–51, 2011.
[5] Shengyi Pan, Thomas Morris, and Uttam Adhikari. De-veloping a hybrid intrusion detection system using datamining for power systems. IEEE Transactions on SmartGrid, 6(6):3104–3113, 2015.
[6] Uttam Adhikari, Thomas Morris, and Shengyi Pan. Wamscyber-physical test bed for power system, cybersecuritystudy, and data mining. IEEE Transactions on Smart Grid,8(6):2744–2753, 2016.
[7] Shiva Poudel, Zhen Ni, and Naresh Malla. Real-timecyber physical system testbed for power system securityand control. International Journal of Electrical Power &Energy Systems, 90:124–133, 2017.
[8] Y. Yang, H. Xu, L. Gao, Y. Yuan, K. McLaughlin, andS. Sezer. Multidimensional intrusion detection system foriec 61850-based scada networks. IEEE Transactions onPower Delivery, 32(2):1068–1078, 2017.
[9] I. N. Fovino, M. Masera, L. Guidi, and G. Carpi. Anexperimental platform for assessing scada vulnerabilitiesand countermeasures in power plants. In 3rd InternationalConference on Human System Interaction, pages 679–686,2010.
[10] I. A. Oyewumi, A. A. Jillepalli, P. Richardson, M. Ashra-fuzzaman, B. K. Johnson, Y. Chakhchoukh, M. A. Haney,F. T. Sheldon, and D. C. de Leon. Isaac: The idaho cpssmart grid cybersecurity testbed. In 2019 IEEE TexasPower and Energy Conference (TPEC), pages 1–6, 2019.
[11] B. Chen, K. L. Butler-Purry, A. Goulart, and D. Kundur.Implementing a real-time cyber-physical system test bed inrtds and opnet. In 2014 North American Power Symposium(NAPS), pages 1–6, 2014.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 15
[12] A. Nelson, S. Chakraborty, Dexin Wang, P. Singh, QiangCui, Liuqing Yang, and S. Suryanarayanan. Cyber-physical test platform for microgrids: Combining hard-ware, hardware-in-the-loop, and network-simulator-in-the-loop. In 2016 IEEE Power and Energy Society GeneralMeeting (PESGM), pages 1–5, 2016.
[13] M. Mallouhi, Y. Al-Nashif, D. Cox, T. Chadaga, andS. Hariri. A testbed for analyzing security of scada controlsystems (tasscs). In ISGT 2011, pages 1–7, 2011.
[14] H. G. Aghamolki, Z. Miao, and L. Fan. A hardware-in-the-loop scada testbed. In 2015 North American PowerSymposium (NAPS), pages 1–6, 2015.
[15] A. Sahu, A. Goulart, and K. Butler-Purry. Modeling aminetwork for real-time simulation in ns-3. In 2016 Princi-ples, Systems and Applications of IP Telecommunications(IPTComm), pages 1–8, 2016.
[16] S. Yafen, C. Jiaxi, Y. Jing, and H. Ning. Reliability analysisof system-in-the-loop network platform based on delays. In2011 Seventh International Conference on ComputationalIntelligence and Security, pages 750–753, 2011.
[17] R. van Heerden, H. Pieterse, I. Burke, and B. Irwin. De-veloping a virtualised testbed environment in preparationfor testing of network based attacks. In 2013 InternationalConference on Adaptive Science and Technology, pages1–8, 2013.
[18] S. Tan, W. Song, Qifen Dong, and L. Tong. Score: Smart-grid common open research emulator. In 2012 IEEE ThirdInternational Conference on Smart Grid Communications(SmartGridComm), pages 282–287, Nov 2012. doi: 10.1109/SmartGridComm.2012.6485997.
[19] V. Venkataramanan, A. Srivastava, and A. Hahn. Real-time co-simulation testbed for microgrid cyber-physicalanalysis. In 2016 Workshop on Modeling and Simulationof Cyber-Physical Energy Systems (MSCPES), pages 1–6,April 2016. doi: 10.1109/MSCPES.2016.7480220.
[20] Elaine M Raybourn, Michael Kunz, David Fritz, and VinceUrias. A zero-entry cyber range environment for futurelearning ecosystems. In Cyber-Physical Systems Security,pages 93–109. Springer, 2018.
[21] Jay Johnson. Sceptre: Power system and networking co-simulation environment, 07 2017.
[22] B. Palmintier, D. Krishnamurthy, P. Top, S. Smith,J. Daily, and J. Fuller. Design of the helics high-performance transmission-distribution-communication-market co-simulation framework. In 2017 Workshop onModeling and Simulation of Cyber-Physical Energy Sys-tems (MSCPES), pages 1–6, 2017.
[23] Junho Hong, Reynaldo Nuqui, Dmitry Ishchenko,Zhenyuan Wang, Tao Cui, Anil Kondabathini, David Coats,and S Kunsman. Cyber-physical security test bed: A plat-form for enabling collaborative cyber defense methods. InPACWorld Americas Conference, 2015.
[24] Ren Liu, Ceeman Vellaithurai, Saugata S Biswas,Thoshitha T Gamage, and Anurag K Srivastava. Analyzingthe cyber-physical impact of cyber events on the powergrid. IEEE Transactions on Smart Grid, 6(5):2444–2453,2015.
[25] Mladen Kezunovic, Ahad Esmailian, Manimaran Govin-darasu, and Ali Mehrizi-Sani. The use of system in theloop, hardware in the loop, and co-modeling of cyber-physical systems in developing and evaluating new smartgrid solutions. In Proceedings of the 50th Hawaii Interna-tional Conference on System Sciences, 2017.
[26] Aditya Ashok, Pengyuan Wang, Matthew Brown, and Man-imaran Govindarasu. Experimental evaluation of cyber at-tacks on automatic generation control using a cps securitytestbed. In 2015 IEEE Power & Energy Society GeneralMeeting, pages 1–5. IEEE, 2015.
[27] Vasileios A Papaspiliotopoulos, George N Korres,Vasilis A Kleftakis, and Nikos D Hatziargyriou. Hardware-in-the-loop design and optimal setting of adaptive pro-tection schemes for distribution systems with distributedgeneration. IEEE Transactions on Power Delivery, 32(1):393–400, 2015.
[28] A. Ashok, Pengyuan Wang, M. Brown, and M. Govin-darasu. Experimental evaluation of cyber attacks on auto-matic generation control using a cps security testbed. In2015 IEEE Power Energy Society General Meeting, pages1–5, 2015.
[29] T. J. Overbye, Z. Mao, A. Birchfield, J. D. Weber, andM. Davis. An Interactive, Stand-Alone and Multi-UserPower System Simulator for the PMU Time Frame. In2019 IEEE Texas Power and Energy Conference (TPEC),pages 1–6, 2019.
[30] Thomas J Overbye, Zeyu Mao, Komal S Shetye, andJames D Weber. An interactive, extensible environmentfor power system simulation on the pmu time frame with acyber security application. In 2017 IEEE Texas Power andEnergy Conference (TPEC), pages 1–6. IEEE, 2017.
[31] Zeyu Mao, Hao Huang, and Katherine Davis. W4ips: Aweb-based interactive power system simulation environ-ment for power system security analysis. In Proceedingsof the 53rd Hawaii International Conference on SystemSciences, 2020.
[32] Jay Johnson, Ron Ablinger, Roland Bruendlinger, Bob Fox,and Jack Flicker. Interconnection standard grid-supportfunction evaluations using an automated hardware-in-the-loop testbed. IEEE Journal of Photovoltaics, 8(2):565–571,2018.
[33] Matsu Thornton, Mahdi Motalleb, Holm Smidt, JohnBranigan, Pierluigi Siano, and Reza Ghorbani. Internet-of-things hardware-in-the-loop simulation architecture forproviding frequency regulation with demand response.IEEE Transactions on Industrial Informatics, 14(11):5020–5028, 2017.
[34] Emilio C Piesciorovsky and Noel N Schulz. Fuse relayadaptive overcurrent protection scheme for microgrid withdistributed generators. IET Generation, Transmission &Distribution, 11(2):540–549, 2017.
[35] T. Becejac, C. Eppinger, A. Ashok, U. Agrawal, andJ. O’Brien. Prime: a real-time cyber-physical systemstestbed: from wide-area monitoring, protection, and con-trol prototyping to operator training and beyond. IETCyber-Physical Systems: Theory Applications, 5(2):186–195, 2020.
Preprint – Design and Evaluation of A Cyber-Physical Resilient Power System Testbed 16
[36] B. Azimian, P. M. Adhikari, L. Vanfretti, and H. Hoosh-yar. Cross-platform comparison of standard power systemcomponents used in real time simulation. In 2019 7thWorkshop on Modeling and Simulation of Cyber-PhysicalEnergy Systems (MSCPES), pages 1–6, 2019.
[37] Matthias Stifter, Jose Cordova, Jawad Kazmi, and RezaArghandeh. Real-time simulation and hardware-in-the-loop testbed for distribution synchrophasor applications.Energies, 11(4):876, 2018.
[38] Y. Yang, K. McLaughlin, T. Littler, S. Sezer, E. G. Im, Z. Q.Yao, B. Pranggono, and H. F. Wang. Man-in-the-middle at-tack test-bed investigating cyber-security vulnerabilities insmart grid scada systems. In International Conference onSustainable Power Generation and Supply (SUPERGEN2012), pages 1–8, 2012.
[39] Arnav Kundu, Abhijeet Sahu, Erchin Serpedin, andKatherine Davis. A3d: Attention-based auto-encoderanomaly detector for false data injection attacks. Elec-tric Power Systems Research, 189:106795, 2020. ISSN0378-7796. doi: https://doi.org/10.1016/j.epsr.2020.106795. URL http://www.sciencedirect.com/science/article/pii/S0378779620305988.
[40] Mete Ozay, Inaki Esnaola, Fatos Tunay Yarman Vural,Sanjeev R Kulkarni, and H Vincent Poor. Machine learn-ing methods for attack detection in the smart grid. IEEETransactions on Neural Networks and Learning Systems,27(8):1773–1786, 2016.
[41] Open dnp3 documentation. https://dnp3.github.io/.
[42] Analysis of the cyber attack on the ukrainian power grid:Defense use case. https://ics.sans.org/media/E-ISAC_SANS_Ukraine_DUC_5.pdf, 2016.
[43] A. B. Birchfield, T. Xu, K. M. Gegner, K. S. Shetye, andT. J. Overbye. Grid structural characteristics as validationcriteria for synthetic networks. IEEE Transactions onPower Systems, 32(4), July 2017. ISSN 0885-8950.
[44] P. Wlazlo, K. Price, C. Veloz, A. Sahu, H. Huang,A. Goulart, K. Davis, and S. Zounouz. A cyber topol-ogy model for the texas 2000 synthetic electric powergrid. In 2019 Principles, Systems and Applications of IPTelecommunications (IPTComm), pages 1–8, 2019.
[45] C. M. Davis, J. E. Tate, H. Okhravi, C. Grier, T. J. Overbye,and D. Nicol. Scada cyber security testbed development.In 2006 38th North American Power Symposium, pages483–488, 2006.
[46] N. Gaudet, A. Sahu, A. E. Goulart, E. Rogers, and K. Davis.Firewall configuration and path analysis for smartgrid net-works. In 2020 IEEE International Workshop TechnicalCommittee on Communications Quality and Reliability(CQR), pages 1–6, 2020.
[47] Hao Huang and Katherine Davis. Extracting substationcyber-physical architecture through intelligent electronicdevices’ data. In 2018 IEEE Texas Power and EnergyConference (TPEC), pages 1–6. IEEE, 2018.
[48] Carla Schroder. Dynamic linux routing with quagga.https://www.linux.com/topic/networking/dynamic-linux-routing-quagga/, 2018.
[51] Samuel East, Jonathan Butts, Mauricio Papa, and SujeetShenoi. A taxonomy of attacks on the dnp3 protocol.volume 311, 03 2009. doi: 10.1007/978-3-642-04798-5_5.
[52] Schweitzer Engineering Laboratories. Inc, Oct 2020. URLhttps://selinc.com/products/3530/.
[53] Angela D. Orebaugh, Simon Biles, and Jacob Babbin.Snort Cookbook. O’Reilly Media, Inc., 2005. ISBN0596007914.
[54] Packetbeat in elk stack. https://www.elastic.co/beats/packetbeat.
[55] Zabbix for network monitoring. https://www.zabbix.com/network_monitoring.
[56] A. P. Ortega, X. E. Marcos, L. D. Chiang, and C. L. Abad.Preventing arp cache poisoning attacks: A proof of conceptusing openwrt. In 2009 Latin American Network Opera-tions and Management Symposium, pages 1–9, 2009. doi:10.1109/LANOMS.2009.5338799.
[57] Mohammad Narimani, Hao Huang, AmarachiUmunnakwe, Zeyu Mao, Abhijeet Sahu, SamanZonouz, and Kate Davis. Generalized contingencyanalysis based on graph theory and line outage distributionfactor, 07 2020.
[58] R. Kalluri, L. Mahendra, R. K. S. Kumar, and G. L. G.Prasad. Simulation and impact analysis of denial-of-service attacks on power scada. In 2016 National PowerSystems Conference (NPSC), pages 1–5, 2016. doi:10.1109/NPSC.2016.7858908.
