Operator Training Simulator (OTS) – Grid Reliability
Sreenivas Badri
Manager, EMMS Production Support
09/25/2019
PUBLIC
Agenda
2
• ERCOT Region, Peak Demand Records
• OTS Environments, Architecture and Data flows
• OTS Case Development Process overview
• New Training Facility
• Operator Training Cycles
• Integrated OTS system capabilities
• Control Room Operators Training – Summer Readiness
PUBLIC
The ERCOT Region
3
The interconnected electrical system serving most of Texas, with limited external connections
• 90% of Texas electric load; 75% of Texas land
• 74,531 MW peak, August 12, 2019
• More than 46,500 miles of transmission lines
• 650+ generation units (excluding PUNs)
ERCOT connections to other grids are limited to ~1,250 MW of direct current (DC) ties, which allow control over flow of electricity
600 MW with SPP
30 MW with CENACE
at Eagle Pass
100 MW with CENACE
at Laredo 300 MW with CENACE at McAllen
220 MW with SPP
Texas RE
PUBLIC
Current Records
Peak Demand Record: 74,531 megawatts (MW)*
August 12, 2019, 4-5 p.m.
Weekend Peak Demand Record: 71,864 MW*
Sunday, August 11, 2019, 5-6 p.m.
Winter Peak Demand Record: 65,915 MW
Jan. 17, 2018, 7-8 a.m.
Wind Generation Records (instantaneous)
Output: 19,672 MW
‒ Jan. 21, 2019, 7:19 p.m.
Penetration (load served): 56.16%
‒ January 19, 2019, 3:10 a.m.
‒ Total MW Served by Wind = 17,406 MW
Recent Monthly Peak Demand Records
2019• March: 60,692 MW (March 5, 7-8 a.m.)
• August: 74,531 MW (August 12, 4-5 p.m.)*
2018• January: 65,915 MW (Jan. 17, 7-8 a.m.)
• May: 67,265 MW (May 29, 4-5 p.m.)
• June: 69,123 MW (June 27, 4-5 p.m.)
• July: 73,473 MW (July 19, 4-5 p.m.)
• November: 56,317 MW (Nov. 14, 7-8 a.m.)
*New records are preliminary, subject to change in final settlement
4
PUBLIC
Operator Training Simulator (OTS) – Environments
5
• 6 OTS Environments, each environment contains multiple systems
• 1 – Test, 2- Bastrop Production and 3- Taylor Production environments
EMS
PI OT04OT03OT02OT01-Test OT05
PI Buffer & Archive
PI Buffer & ArchiveWebFG/ETA
DSA/VSAT/TSAT
MMSMA App
MMSMOI Web
EMS
Webfg/ETA
DSA/VSAT/TSAT
MMSMA App
MMSMOI Web
EMS
Webfg/ETA
DSA/VSAT/TSAT
MMSMA App
MMSMOI Web
EMS
Webfg/ETA
DSA/VSAT/TSAT
MMSMA App
MMSMOI Web
EMS
Webfg/ETA
DSA/VSAT/TSAT
MMSMA App
MMSMOI Web
PI AF/Coresight
PI AF/Coresight MMS LDAP
LDAP Servers
PI Buffer & Archive
OT06
EMS
Webfg/ETA
DSA/VSAT/TSAT
MMSMA App
MMSMOI Web
PI AF/Coresight
EMS & MKT Oracle Databases
EMS & MKT Oracle Databases
EMS& MKT Oracle Database
EMS & MKT Oracle Databases
EMS & MKT Oracle Databases
EMS & MKT Oracle Databases
MMSMI/OS
MMSMI/OS
MMSMI/OS
MMSMI/OS
MMSMI/OS
MMSMI/OS
MMAP MMAP MMAP MMAP MMAP MMAP
PUBLIC
Operator Training Simulator (OTS) – Environments
6
o To have flexibility to run 2-3 simulations in parallel in a training cycles
o To troubleshoot any existing training cycle simulator issues
o To develop and prepare next training cycle simulation
o To train control room operators on latest software release changes before
the changes hits production
• All OTS Production environments (except being used for current Training Cycles)
OTS systems & applications software release versions are in sync with actual
production versions.
• OTS environments are treated as critical with production level support and
maintenance considering the significance of Operator Training and back to back
training cycles is being planned throughout the year.
• Test OTS environment is used for OTS software changes testing before deploying
into Production OTS environments
• Why we need 5 Production OTS environments ?
PUBLIC
OTS Systems Architecture
7
EMS & OTS Applications
Data Interface
OTS EMS System - Linux Server
E-terraArchiver Application
Windows Server
EMS
Ora
cle
Dat
abas
e (A
IX
Serv
er)
MM
S O
racl
e D
atab
ase
(AIX
Se
rver
)
Market Operator Interface (MOI) – Display
Server
MMS Application Server(SCED, HRUC, ASM, VDI
etc.)
MMS/OS MI Server(Market/Outage
Scheduler Events)
Tibco
OTS MMS System - AIX
DSA/VSAT - Windows Server
MMAP Windows Sever
PI Buffer & Archive – Windows Server
WebFG Display Server
PI AF/Coresight – Windows Server
Web Based PI Ops Tools
PUBLIC
OTS Applications Data flows
8
DSA/VSAT/TSAT
RTCA
RTNET
QKNET
EMS - Network Applications
TCM SCED
Market Management System
SCADA
LF
RLC
LFC
DTSPSM QSE Simulator
RTOP
Data Interface
EMS/OTS HDR
Event Scenarios
EMS - Generation Applications
OTS - Simulation Engine
MOI Displays – COP/DCTIE User Interface
COP/DCTIE/RUC/ERS/AS Deployments/Outages
(MMS Database)
Outage Scheduler(MMS Database)
HRUC/AS Manager/SASM
MOI Displays – Outages creation/
Update- User Interface
STNET
DTSPSM also mimics TO Role & TO TelemetryQSE Simulator mimics QSE Role & QSE Telemetry
iGridInterface(Reliability Risk
Desk Operations)
RDB Interface
Wind & Solar Forecast
(EMS Database)
OTS Events
PUBLIC
OTS Case Development Process Overview
9
• OTS case development process involves multiple teams, most of the tasks are being
performed by EMMS Production and Operator Training Teams
• End to End Case development, all OTS systems setup takes around 2 weeks
depending on number of simulations are planned in a given training cycle.
• Training Team provides following training cycle requirements and determines new OTS
case is required.
Production Snapshot Date, Peak Demand
Systems and Applications, Tools to be functional
Market Simulation requirements
• EMMS Production develops OTS case, performs following testing on the validated
case to ensure simulator is working as expected before adding OTS events. OTS (DTSPSM, QSE Simulator etc.) applications testing
EMS & MMS application testing and MMS to EMS integration testing
Basic scenario testing (Frequency Recovery, Large Unit Trips, 345KV equipment outages, RRS/Non-
Spin Deployments etc.)
• Tested case is provided to Training Team to develop OTS events and perform case
tuning to simulate required events during operator training cycles.
PUBLIC
OTS Case Development Process Overview
10
High level steps in OTS Case Development• EMS save cases, SCADA HDR files preparation and setup based on Simulation Date and time.
• Run OTS Save Case Preparation Application to do following
o EMS Save Cases load
o Validate all Model Clones
o Load validated save cases into Real Time Clones
• Case updates and tuning – run scripts to populate OTS specific data and set OTS specific parameters
• Bring up OTS system and Verify OTS applications specifically QSE Simulator and DTSPSM
• Ensure that DSA/VSAT runs and transfer Stability Limits to EMS
• Perform Market Database Refresh based on Simulation date and time.
• Perform CIM Model Load in Market System based on Simulation date
• Setup EMS/MMS Interfaces and ensure EMS to MMS dataflow is functional
• Enable OTS Market Integration, ensure Market Management System (MMS) applications SCED/HRUC/ASM/VDI
are working as expected based on basic scenario testing.
• Perform OTS Integration Testing
• Prepare PI data and ensure PI Archive, PI AF/CoreSite, PI Ops Tools, Control Room Spreadsheets are functional
based on Simulation time.
• Ensure Macomber Map is setup and integrated with EMS.
PUBLIC
OTS Case Development Lifecycle
11
Simulation Case Development Lifecycle
EM
MS
Pro
du
ctio
n S
upp
ort
Sys
tem
Ope
rati
ons
Tra
inin
gE
MM
S
De
velo
pm
ent
Da
tab
ase
Ho
stin
g
Ser
vice
s
Sys
tem
Op
era
tions
Overall development timeline depends on accurate estimate of required work in each department
Start
Complete
Simulation Case
Preparation
Checklist and
submit to EMMS
Test case Fixes required?
Evaluate case
Simulation
performance
acceptable?
No
RFC for code
changes
required?
Create code or
tuning changes
and deliver to
EMMS PS
No
Event
performance
acceptable?
Yes Yes
Yes
Sim dev
process
efficiencies
identified?
Yes
Update Simulation
Case Preparation
Checklist
End
Highlighted boxes represent individual department-specific documented processes/procedures/controls which require
department commitments
Red arrows represent the optimal path
Training/
Performance
simulation
requested
Create simulation
training script
Ops
procedures/
tool issues
identified?
Submit to
Operations
Improvement for
analysis
No
No
Yes
YesCreate RFC for
Dev
Conduct training
and support
training
Simulation
indications
acceptable?
Yes
NoDevelop case
Yes
Perform Market
data refresh
Market data
refresh req’d?No
No
No
Create crew
performance
feedback report
PUBLIC
Taylor Training Facility
12
• Black Start and Operator
Training Seminars will
take place in the new
building in 2019.
• ERCOT’s new training center
officially opened in January.
PUBLIC
Operator Training Facility
13
• Operator Training Facility Control Room is replica of actual Control
Room
• Operators from every desk are trained in OTS environment on all the
EMS/VSAT/MMS/PI/MMAP etc. applications and situational awareness
tools they use in actual control room.
PUBLIC
Operator Training Cycles
14
• 6 Operator Training Cycles every year
Black Start Training with Market Participants every year - Cycle 2 (March – April) –
800 Operators (including ERCOT Operators) attending this year Black Start
training this year.
Summer Readiness Training Cycles – Cycle 3 and Cycle 4 (May to August) It also includes SOLs & IROLs, EEA and Loss of Primary Control Center related
training simulations
Operator Training Seminar – Cycle 5 (OTS is not used)
Emergency Weather Drills (Alternate between Hurricane and Winter Storm) –
Cycle 6 (October – November)
Grid Ex – Cycle 6
• Operators are trained in OTS environment on EMS/MMS/PI Ops Tools, Situational
Awareness tools changes that go into production that impacts control room
operations, it includes defects, enhancements and functionality upgrade &
technology upgrade projects.
• Each Training Cycle duration is 6 weeks to train 6 operator crews
PUBLIC
Integrated OTS System Capabilities
15
• Integrated OTS system has capability to simulate various type
of simulations
Simulate Grid Events or EMMS Applications failure scenarios (ICCP/SCADA Telemetry failures,
State Estimator, CA, LFC/RLC/SCED, VSAT etc. failures) and Applications functionality
changes (Nodal Protocol changes, System changes, Enhancements etc.)
Replay Grid and Real-Time Market Events
Standalone mode without Market Integration
Black Start Training with Market Participants
Market Participants Training (Winter Storm Drill, Hurricane Drill)
Simulation based on Future Models (60, 90 days ahead CIM Models)
• Integrated OTS system has capability to run simulation for 48 hours or
more in integration with all other systems including Market System.
PUBLIC
Operator Training – Summer Readiness
16
• Developed OTS case based on 2019 March Production EMS Snapshot and Market
data, overlaid summer peak load pattern.
• OTS Market System data (AS Qualification, COP, Offers) was prepared to setup Market
applications to run SCED/HRUC, deploy Offline Non-Spin, Load Resources RRS, ERS
and dispatch switchable generators during EEA event.
• Training team added events and tuned OTS case to simulate following events to train
Operators for summer events
Energy Emergency Alert (EEA)
Switchable Generators
Real-Time Assessment (RTA)
IROLs and SOLs Exceedances
• All required OTS systems and application were setup to run simulator and simulate
above events during Cycle 3 training cycle (May – June).
PUBLIC
Operator Training – EEA
17
• EEA Trigger Conditions
EEA1 - must be declared when PRC < 2300 MW & it is not projected to be recovered above
2300 MW within 30 minutes
EEA 2 - must be declared when PRC < 1750 MW & & it is not projected to be recovered above
1750 within 30 minutes or may be declared EEA 2 if System frequency below 59.91 for 15
minutes
EEA3 - must be declared when PRC <= 1375 or must go to EEA3 if system frequency is below
59.91 for 25 minutes or may declare EEA3 if system frequency is below 59.91 for 20 minutes
• Energy Emergency Alert (EEA) Days
EEA days this year – 08/13/2019 & 08/15/2019
Last EEA Days – 01/06/2014 & 01/18/2014
• Constant EEA training is required for operators in Operator Training
Simulation environment because
EEA is rare event
EEA situational awareness tools, EMS/MMS applications, processes
and procedures changes constantly
Every operator crew requires to go through real-time EEA events
simulation
o ERS, Offline Non-Spin and Load Resources RRS deployment related MMS
Applications are used mainly during EEA events other than qualification
testing.
PUBLIC
Operator Training – EEA
18
• Operators followed the EEA procedure to maintain Grid reliability during EEA and
brought back the Grid to normal conditions. Following are some of the key actions
performed by Operators during EEA event using OTS EMS and MMS applications
Manual RRS Deployment through LFC
Offline Non-Spin Deployment
ERS Deployments
Load Resource RRS deployment
RUC Commitments through VDI
Firm Load shed through OTS Events
• All Operator crews were trained on EEA event in OTS environment by end of June that
includes all the situational awareness tools changes and EEA event
processes/procedures.
• Operators were also trained on Market System Emergency Response Service (ERS)
and Non-Spin Display enhancements during EEA event training in OTS environment.
• Control Room Shift Supervisors and Operators were extremely happy on how OTS
training on EEA helped them to handle this summer EEA events effectively.
PUBLIC
Operator Training – EEA Dashboard
19
PUBLIC
Operator Training – Switchable Generators
20
• Switchable Generators (SWGRs) changes were brought into Production EMS and MMS
systems end of May this year in preparation to handle peak demand this summer.
• A new telemetered resource status code EMRSWGR is made available to represent
Emergency Switchable Generation resources.
• When Switchable Generation qualified resources are switched from other region and
ready to operate in ERCOT region, QSE changes resource status telemetry from
EMRSWGR to ON, SCED dispatches the units.
• OTS Market System data was prepared to simulate Switchable Generators (SWGRs)
following behavior in OTS environment the way it happens in real-time
Showing SWGRs current and future operating status, capacity in Market System
new Displays
SWGRs Resource Status Telemetry changes in EMS
SCED dispatching SWGR units
• Operators were trained on Switchable Generators operations in OTS environment before
changes were deployed in Production before start of this summer.
• Operators were able perform Switchable Generators Operations effectively during this
summer EEA days.
PUBLIC
Operator Training - RTA Failure Scenarios
21
RTA failure scenarios
Partial Loss of ICCP
Complete Loss of ICCP
Loss of an EMS
Application(RTNET/
RTCA/RTDCP)
Loss of Real Time (RT)
EMS System
Loss of RT & Study (ST)
EMS systems
Perform RTA using
Powerworld / Offline VSAT
using last successful RTA
solution exported in
the applicable format
Use Study EMS system with
last successful RTA solution
available
Perform Manual RTA
Using Enhanced
STNET Online Powerflow in
RT EMS
Perform Manual RTA
Using Enhanced
STNET Online Powerflow in
RT EMS
Use SCADA/RTNET
enhancements to help solve
RTNET
• RTA (Real-Time Assessment) improvements that were put in place in EMS after past
incidents helped significantly ERCOT RTA process running through any TSP telemetry
failures.
• Rigorous Shift Engineer and Operator Training was provided during this summer in OTS
environment by simulating RTA failure scenarios that helped them significantly in
understanding the RTA details and performing manual RTA within 30 minutes.
PUBLIC
Questions?
22
PUBLIC
Thank You
23