FEP Diversity Project

RTS System Admin Group

RTUs, Communications

Real Time Systems2

Agenda

1. Substation – EMS Communications Overview

2. Traditional 4-wire communications

3. FEP Diversity Projecta. Motivators

b. Pilot Project - POC

c. Technical Solutionsi. Communications Architecture

ii. AREVA Database Modifications/Changes

d. Additional Benefits

4. Future Direction – Expansion to other Substation needs

Real Time Systems3

Substation – EMS Communications Overview

Real Time Systems4

Substation to EMS data flow

Operating SystemUnix/ Windows 2000

Real-Time Control System Environment

Energy Management System Applications

HABITAT NETIO Web-FGUtilities: Configuration Manager MRS Alarm

SCADASubsystem

GenerationSubsystem

NetworkSubsystem

Analogs & Statuses

External unit MW schedules,pump unit limits,

incremental cost curves

Frequency, time,plant & tie-line

measurements,unit controls,

ACE, system load

Unit security limits,loss sensitivities,AGC participation

factors

Estimated Values

Unit Controls

FEP

Analogs

StatusesControls

RTUs

Accumulators

SMP – SCADA ManagementPlatform

Substation

Real Time Systems5

SCADA protocols

Protocol is a way of organizing and presenting data in a consistent, repeatable format

Bit and byte oriented protocols– Byte - Harris, DNP– Bit - Conitel, CDC2, TRW S9000, TRW S9550– Special interfaces - Intrac and MOSCAD radio systems

All master station SCADA protocols do the same basic functions

– analog data conversion– status point monitoring– support “select before operate” function for controls– provide analog output capability– provide different types of scans to capture transient events between scans

and minimize bandwidth usage– provide some form of “cyclic redundancy check” CRC so that if the received

message has been corrupted during transport it will be rejected.

Real Time Systems6

FE RTU numbers and types

Approximately 2000 RTUs on FE system

650 in west, 1350 in the east and growing

Diverse hardware platforms

Diverse protocols – including:– DNP 3.0– Harris 5000– CDC II– TRW S9000 / 9550– Conitel 300/3000– Modbus– Intrac/Moscad

Real Time Systems7

RTU communications

Primarily four wire leased circuit – 1200 baud modem

Radio used extensively in east ~ 650 radio equipped RTUs– Primarily found at line switch controllers

Dialup - Approximately 100 dialup RTUs in west– Primarily found at line switch controllers

Real Time Systems8

Substation to EMS data flow

Operating SystemUnix/ Windows 2000

Real-Time Control System Environment

Energy Management System Applications

HABITAT NETIO Web-FGUtilities: Configuration Manager MRS Alarm

SCADASubsystem

GenerationSubsystem

NetworkSubsystem

Analogs & Statuses

External unit MW schedules,pump unit limits,

incremental cost curves

Frequency, time,plant & tie-line

measurements,unit controls,

ACE, system load

Unit security limits,loss sensitivities,AGC participation

factors

Estimated Values

Unit Controls

FEP

Analogs

StatusesControls

RTUs

Accumulators

SMP – SCADA ManagementPlatform

Substation

Real Time Systems9

What is a FEP?

Windows 2k3 server OS

Standard Infrastructure server Hardware– One exception, the FEP has additional Digi PCI cards so that

server has 128 COM ports

Runs the AREVA e_terraControl suite of applications– Process Starter– CFEReader – Polls RTU’s– SCADA – Contains the database– ISD – Connects the FEP to the EMS– Ancillary processes:

– AlarmPager (provides method to pass FEP alarms to EMP)– SpaceSaver– DBUpdate

Key: No FEP’s = No Data entering the EMS

Real Time Systems10

Real Time Systems11

Traditional 4-wire communications

Real Time Systems12

Traditional 4-Wire Communications

Real Time Systems13

FEP Diversity Project

Real Time Systems14

Project Motivators

Issue identified by NERC as a disaster recovery limitation

Loss of DataCenter, either from a physical or communications perspective, essentially eliminates the EMS system.

Upgrade the method of communications to critical transmission substation RTU’s.

Network change without causing system software changes– Keep serial connected FEP ports and serial connected

RTUs

Improve Communications Network Reliability

Real Time Systems15

Proposal to operate selected critical transmission RTU’s from physically diverse FEP’s

3rd/4th Quarter 2005 performed R&D proof of concept testing

Production deployment began 1st quarter of 2006

Initial sites include interconnection and critical transmission sites

By the end of 2006 we successfully cut over 54 production sites

By the end of 2007 we successfully cut over 104 production sites – All 345 and 138kV Tx sites in FE’s MISO area

111 Transmission sites in PJM are scheduled for completion though 2012

All new FE RTU sites are being installed on this technology

Project Overview

Real Time Systems16

Pilot Objectives - What we wanted to prove? Demonstrate that the AREVA system can support data

acquisition using diversified FEP’s at the same level of reliability as currently observed with co-located FEP pairs.

Verify the proposed network communications architecture will support geographically distributed FEP’s.

Demonstrate that in using diversified FEP’s there is no degradation in AREVA system level performance (FEP and EMP).

Demonstrate that alarm processing performance is equivalent to that found with legacy designs - throughput.

Verify system responds properly to the loss of inter-FEP heartbeat.

Real Time Systems17

Required work activities in R&D Pilot period

Build new servers (Windows and Unix)

Develop a point lists for each RTU

Develop software configuration for each RTU

Develop a strategy for modeler configurations

Develop a standard network hardware configuration

Configure AREVA Systems

Configure, debug, and test pilot system

Develop Cut-over strategies for production

Real Time Systems18

Measure of success in all pilot phases? Burn in test using all RTU’s for a one week period of local

continuous polling is successful – No FEP process shutdowns

– Minimal missed scans

Alarm propagation and data integrity - 24 hour test– All status points change state at least once per scan interval

– All analog points change each scan interval

– No missed scans or missed alarms

System stability is maintained for following failover scenarios:– Loss of one FEP

– Loss of inter FEP heartbeat

– Loss of one EMS server

Real Time Systems19

FEP Diversity Pilot Phases

Phase 1 - 10/3/05-10/13/05– Performed local proof of concept testing using new FEP’s and 55

logical RTU’s. Using DYMEC supplied network hardware, failure analysis was be conducted and logical RTU’s polling statistics were gathered. FRAME network is all mocked up locally at DataCenter B.

– End Result- Satisfactory test results based on operational performance and failure analysis. All testing performed locally at DataCenter B.

Real Time Systems20

FEP Diversity Pilot Phases -CONT.

Phase 2 - 10/14/05-10/25/05– DYMEC Equipment was extended to DataCenter A to allow

controlled diversified FEP testing. Required relocating a FEP server to DataCenter A, and the associated FRAME network connectivity.

– End Result- Satisfactory test results based on operational performance and failure analysis.

Real Time Systems21

FEP Diversity Pilot Phases - CONT.

Phase 3 -10/26/05-11/16/05– Selected substations (6 site’s) will be dual ported and connected to

digital circuits. Digital FRAME circuits will appear at DataCenter A and DataCenter B where the diversified FEP’s and EMS servers reside.

– Includes site’s on the FE, and the Telco (leased) FRAME network’s

– End Result- Satisfactory test results based on operational performance and failure analysis.

Real Time Systems22

Results of Pilot

Project team successfully installed and configured the new versions of the SMP software on FEP pair.

– Latest Windows O/S

– Latest AREVA SMP Software

– New hardware components

Verified successful RTU communications using current serial communications.

All three phases of RTU testing completed successfully, including phase III’s real-life field tests

Developed Cutover Plan

Real Time Systems23

Production Implementation Project

Order communication circuits as required– A majority of circuits require High Voltage Protection (Positron)

Make necessary RTU configuration changes– Baud Rate / Modem settings

Complete AREVA modeler work to allow cutover to occur– Run ManageSiteAssignments.pl script to use Primary/Secondary SITE ID– Make PATH changes for new communication circuit

Cutover RTU’s to Diversified FEP– Perform full station check-out on new circuit

Real Time Systems24

Project Timeline

Real Time Systems25

FEP Diversity Architecture

Real Time Systems26

FILENAME

TITLE

FEP DIVERSIFICATION PILOT PHASE #1

Network Services EngineeringConfidential

PAGE

1 OF 1

REVISED

9/14/2006

FULL FILENAME

DRAWN BY

REX DUDAS

FEPDIVERSIFICATION

DYMEC(Physical)

DYMEC

- DSU/CSU (Digital Circuit Interface)(T1-Fractional, 64k)

- ROUTER

-FRAD (Frame-Relay Access Device)

-SUBSTATION HARDENED(-40C -+75C) no fans

5 PORT ETHERNET SWITCH(IP ONLY DEVICES i.e.: PC,

IP CAMERA...etc)

8 PORT TERMINAL SERVER(SERIEL CONNECTED

DEVICES i.e: RTU, DFR,LINE SWITHCES...etc)

FRAME RELAYNETWORK

- BUILT-IN DATA ANALYZER

- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL

10/100

max.: 38400bps

Real Time Systems27

FILENAME

TITLE

FEP DIVERSIFICATION PILOT PHASE #1

Network Services EngineeringConfidential

PAGE

1 OF 1

REVISED

9/14/2006

FULL FILENAME

DRAWN BY

REX DUDAS

FEPDIVERSIFICATION

DYMEC(LOGICAL)

(ROUTER)

5 PORT ETHERNET SWITCH(IP ONLY DEVICES i.e.: PC,

IP CAMERA...etc)

8 PORT TERMINAL SERVER(SERIEL CONNECTED

DEVICES i.e: RTU, DFR,LINE SWITHCES...etc)

- FRAME/SERIAL ENCAPSULATION RTU PROTOCOL(NON-ROUTABLE, ENCRYPTED)

RTU

DYMEC

FRAME RELAYNETWORK PVC 1

PVC 2

PVC 3

CORP. TRAFFIC(FIREWALL, ENCRYPTED)

DataCenter A FEPA

DataCenter B FEPB

RTU MAINTENANCE PORT

IP ROUTABLE PROTOCOL REMOVED OVER WANLINK

Real Time Systems28

Cut-over Strategy (part1)

Need to have both existing and diversified RTU modeled for cutover in AREVA EMS– Model an additional TFE structure for new diversified FEP

– Changes to BAUD, PTMO1, PTMO2 and NoReplyTimeout– Additional RTUC Record pointing to same RTU record as 4-wire circuit

in production– Utilize SITE_* and SITE2_* fields to facilitate the cutover

– Each RTU has potentially hundreds of POINT/ANALOG/COUNT records associated with it.

– These all need their SITE_* field set to the ID_SITE of the new diverse FEP.

– SITE2_* field set to the ID_SITE of the current 4-wire FEP in production– Manage those individual data items that do NOT allow automatic

SITE/SITE2 selection. – Selection is done automatically by SCANNER based on quality. Items

with no status, (CTRL only) do not have a good quality code to follow (TAP Changers)

Areva TFE tree changes made, No RTU tree changes made.– Limit the risk of the change – No production affects

Real Time Systems29

Cut-over Strategy (part2)

RTU Changes– New FRAME T1 Circuit can be installed at any time on

spare pairs– Make RTU configuration changes and load onto spare D20

processor board– Swap out board for cutover, if there are issues original board

can be immediately put back

– No Point mapping changes made

Real Time Systems30

POST DESIGN BENIFITS DR Environment

– Remote RTU can be pulled from either DataCenter Eliminates costly – Out-of-date Analog 4-wire circuits Increase RTU speed/poll interval

– 1200bps to 9600bps (max.: 38400 bps)

Ability to pull byte and bit-oriented RTUs Remote Support

– Allow for mobile workforce support/configuration/data analysis – Secured IP Based

Security– Serial Connectivity – Isolation of corporate substation traffic

HP/OV Support– Complete Network end-to-end alarming

“Smart” Substation platform– Ability to add additional PVC’s for other needs

– IP enabled Security Cameras for monitoring site– Swipe card entry systems

CIP Cyber Security full compliance– Non-Routable Transport

Real Time Systems31

FEP Diversification Overview

Real Time Systems32

FE

Cor

p N

etw

ork

RP

R B

ackb

one

FE Isolated WANSONET/RPR

EMS SONET/RPRBackbone

T1T1

Substation RTU NetworkRPR Exchange WAN

BG #TBD

GMS SONET/RPRBackbone

RTS FWDC-A

RTS FWDC-B

EMS FEPsDC-A

Sub-RTRDC-A

Sub-RTRDC-B

Public CarrierFR/MPLS

TCP/IPNetwork

AggregateRTU Router

Site #1

SubstationRTU Router

Site #5

AggregateRTU Router

Site #4

AggregateRTU Router

Site #3

AggregateRTU Router

Site #2

SubstationRTU

Site #4

SubstationRTU

Site #5

SubstationRTU

Site #3Substation

RTUSite #2

SubstationRTU

Site #1

DynamicFailover

FirstEnergy FEP Diversity: Proposed IP Based Network Architecture A

DynamicFailover

Sub-RTRDC-C

RTS FWDC-C

DynamicFailover

DynamicFailover

EMS FEPsDC-B

EMS FEPsDC-C

GMS DACsDC-A

GMS DACsDC-B

T1

Fa

stE to

ML C

ard

FastE to

ML Card

FastE to

ML Card

Consolidate RTU Communication over anIP Based FE RPR Wan and/or Public

Carrier Frame/MPLS Networks.

Corp FWDC-A

Corp FWDC-D

Electronic Security Perimeter

SubstationRTU

Site #1a

SubstationRTU

Site #1b

Fractional T1

Fa

stE to

ML C

ard

Fast

E t

oM

L C

ard

Ele

ctro

nic

Se

curit

y Pe

rime

ter

Real Time Systems33

EMS SONET/RPRBackbone

FE

Cor

p N

etw

ork

RP

R B

ackb

one

FE Isolated WANSONET/RPR

T1T1

Substation RTU NetworkRPR Exchange WAN

BG #TBD

GMS SONET/RPRBackbone

RTS FWDC-A

RTS FWDC-B

Sub-RTRDC-A

Sub-RTRDC-B

Public CarrierFR/MPLS

TCP/IPNetwork

AggregateRTU Router

Site #1

SubstationRTU Router

Site #5

AggregateRTU Router

Site #4

AggregateRTU Router

Site #3

AggregateRTU Router

Site #2

SubstationRTU

Site #4

SubstationRTU

Site #5

SubstationRTU

Site #3Substation

RTUSite #2

SubstationRTU

Site #1

DynamicFailover

FirstEnergy FEP Diversity: Proposed IP Based Network Architecture B

DynamicFailover

Sub-RTRDC-C

RTS FWDC-C

DynamicFailover

DynamicFailover

GMS DACsDC-A

GMS DACsDC-B

T1

Fa

stE to

ML C

ard

FastE to

ML Card

FastE to

ML Card

Consolidate RTU Communication over anIP Based FE RPR Wan and/or Public

Carrier Frame/MPLS Networks.

Corp FWDC-A

Corp FWDC-D

Electronic Security Perimeter

SubstationRTU

Site #1a

SubstationRTU

Site #1b

Fractional T1

Fa

stE to

ML C

ard

Fast

E t

oM

L C

ard

Ele

ctro

nic

Se

curit

y Pe

rime

ter

DYMECDYMECDYMEC

EMS FEPsDC-A

EMS FEPsDC-B

EMS FEPsDC-C

Real Time Systems34

Questions?

??