Why Industry Needs Time A Power Industry Case Study
Moderator
Lloyd Green, Director of Engagement Marketing & Creative
Community Services, IEEE-SA
Panelists
Anand Ram, Vice President Marketing & Sales, Calnex
Aaron Martin, Power System Protection Engineer, BPA
Jim Olsen, Director, North American Solutions Architect, Microsemi
1
ATIS Board of Directors’ Meeting October 20, 2011
Why Industry Needs Time Webinar May 4, 2016
Smart Grid: Greater Complexity
3
Source: Calnex Solutions
From centralized generation, distribution and transmission in one direction . . .
to renewables, distributed generation, real-time billing, improved control
High synchronization requirements due to distributed nature of the Smart Grid and the critical balance between power generation and consumption
• Power can’t be stored easily so grids generate according to demand
• Need good comms and sync to correlate demand and generation
ATIS Board of Directors’ Meeting October 20, 2011
Why Industry Needs Time Webinar May 4, 2016
Smart Grid: Driving Data
4
Data gathering and distribution in real-time is becoming
increasingly important
Source: Calnex Solutions
Information must be time specific (needs Time of Day and Phase information)
Data must be shared between remote locations (increased throughput)
Synchronization
ATIS Board of Directors’ Meeting October 20, 2011
Why Industry Needs Time Webinar May 4, 2016
The Need for Synchronization in Power Systems
5
Many applications are enabled/improved by precise synchronization:
• IEDs for Power System Automation require time/phase information (protection, instrumentation and control, etc.)
• Real-time billing
• Enhanced differential protection (increasingly important as distributed generation grows)
Source: Calnex Solutions
Different applications, vendors, requirements
Deployment challenges
Anand Ram, Vice President Marketing & Sales
Calnex Solutions
Aaron Martin, Power System Protection Engineer
Bonneville Power Administration
Jim Olsen, Director, North American Solutions Architect
Microsemi
Today’s Speakers
Company Confidential
Company Confidential
Mobile Backhaul
Power Generation
Automotive, Broadcast, etc.
Industrial Internet
Financial Markets
Company Confidential
IEEE 1588 states in clause 19.3.1.1: "The purpose of a PTP profile is to allow organizations to specify specific selections of attribute values and optional features of PTP that, when using the same transport protocol, inter-work and achieve a performance that meets the requirements of a particular application."
Company Confidential
Mobile Backhaul
PTP with No Timing Support (G.8265.1)
Company Confidential
Mobile Backhaul
PTP with Full Timing Support (G.8275.1)
Company Confidential
Mobile Backhaul
PTP with Assisted Partial Timing Support
New Power Generation: the SMARTGRID
Company Confidential
Greater complexity and diversity drives the need for better sync
Company Confidential
PTP = greater efficiencies + greater diversity Power Profile IEEE C37.238 - being revised as a ‘Level 2’ profile: additional features built on top of ‘Level 1’ Utility Profile IEEE/IEC 61850-9-3
Power Profile IEEE C37.238-2011 currently being aligned with new IEEE 1588
Financial Markets High-Frequency Trading requires accurate timestamping of trades
Company Confidential
Power Profile IEEE C37.238-2011 currently being aligned with new IEEE 1588
But network synchronization is not guaranteed
Company Confidential
Company Confidential
The solution? Timing via 1588 PTP
IETF: Draft Enterprise Profile for PTP (latest version -06, Jan. 2016)
Industrial Internet Time Sensitive Networking
Company Confidential
More and more objects are becoming embedded with sensors and actuators - and gaining the ability to communicate.
Company Confidential
which means ... standardization of communications protocols - and specifically PTP profiles - will be required
Broadcast equipment becoming increasingly connected via the Ethernet
Company Confidential
SMPTE 2059-2: “SMPTE Profile for Use of IEEE1588 Precision Time Protocol in Professional Broadcast Applications” was approved in March 2015
From modest aspirations ...
Company Confidential
SMPTE standardizing on use of PTP for synchronization
Draft ST 2059-2: “Precision Time Protocol SMPTE profile” has gone to ballot
Now, Ethernet is keeping the world in sync.
Bonneville Power Adminstration
Aaron Martin – System Protection Engineer
BPA SERVICE TERRITORY
BPA SERVICE TERRITORY
Power System Applications that
Require Precise Time
Phasor Measurement Units
Line Differential Protection
Traveling Wave Fault Location
System Event Recordings
Substation Local Area Networks
PHASOR MEASURMENT UNITS or
SYNCROPHASORS
PMUs Synchro-phasors, or Phasor
Measurement Units (PMUs), are synchronized
measurement systems that provide information
on phasor angles at different power system
locations.
WISP – Western Interconnection
Synchrophasor Program
PHASOR MEASURMENT UNITS or
SYNCROPHASORS
PHASOR MEASURMENT UNITS or
SYNCROPHASORS
POWER SYSTEM TIMING REQUIRMENTS
Why PMUs
High speed, real time data stream
60 samples per second
Time synchronized measurements
Wide area phase angle differences
Flexibility in data stream
Analog and digital values are included
August 2003 East Coast Blackout
PHASOR MEASURMENT UNITS or
SYNCROPHASORS
PMU vs. SCADA
PHASOR MEASURMENT UNITS or
SYNCROPHASORS
Abnormal Angle
31
Abnormal Angle
32
Frequency Event Location
33
Line Differential Relays
Compare current magnitudes, phase angles to two ends of a
transmission line
Communicate information between relays
Internal/external fault? Trip/no trip?
Communications and high accuracy time dependent!
Changes in communications paths, channel delays, or timing
errors can cause potential problems
Traveling Wave Fault Location
Drop a stone in a still pond
you produce a wave that moves out from the center
In ever increasing circles
Traveling Wave Fault Location
Requires a sampling rate of 1.5 Mhz (500 Nano-seconds) for
fault locations to be accurate within 500 feet.
Traveling Wave Fault Location
Traveling Wave Event
System Event Recording
Digital Events requires millisecond time-tagging
Equipment Alarms, Power Circuit Breaker Operations
Substation Local Area Networks IEC - 61850 Applications
IEC – 61850 GOOSE
Messages – 1ms accuracy
SNTP – 1ms accuracy
IEC – 61850 Sample Values
1 micro second
accuracy
1PPS Sync Source
reliant on C37.238 /
61850-9-3 PTP clocks
Power System Disturbances Caused by
Un-Synchronized Time
500kV transmission line outage
caused by Bad GNSS data
Investigation revealed GPS
satellite testing
GPS receivers within IED did not
meet Air Force GPS standard
ignored GPS Test bit
Power System Disturbances Caused by
Un-Synchronized Time
Line current differential during GPS testing
Power System Disturbances Caused by
Un-Synchronized Time
Resolutions and joint learning
Track planned GPS testing notices
Use GPS receivers that detect test
mode
Use in-channel timing as
source/backup
Desensitized differential phase
Example of PMU Bad Data Caused by
Un-Synchronized Time
Several PMUs lost GPS Sync on 3/15 for
several minutes
During the event, there were multiple
instances of both PMUs reporting that the
synch had been re-established and that their
status was good - even though it wasn't.
Redundant PMUs did not experience this
problem.
Example of PMU Bad Data Caused by
Un-Synchronized Time
We are confident that the synch was not re-
established because the phase angle for these
PMUs jumped by 40 degrees each time the status
cleared.
BPA Communications
BPA covers 4 states with thousands of
command and control circuits on legacy
systems.
The legacy communications systems are
going EOL
Maintenance is also becoming an issue
What do we have?
The BPA communication system consists of Microwave (MW) and
Synchronous Optical Networking (SONET) over Fiber Optics (FO). All of
these communications systems utilize Time Division Multiplexing (TDM)
technology to share the total bandwidth. Though functional, this
implementation is inefficient and costly when supporting IP networks.
SONET Fiber
or
Any Microwave
NMS NMS
FIN FIN
Phasors Phasors
0 1
31
3
4
5
GenICCP GenICCP
Security
Relays
Security
$26K
$26K
$26K
$26K
$26K
$26K Relays
What do we need?
We need a system with the reliability of SONET but the ability to make use of
idle bandwidth for other applications. We need traffic to travel across the
transport at the maximum speed the physical medium supports, to allow new
applications of technology. We need a system requiring less overall impact.
SONET Fiber
or
Any Microwave
NMS NMS
FIN FIN
Phasors Phasors
Video Video
Video conferencing needs 384 kbps.
Assume 4 hours use per day = 5.5 Gbit.
20 hours of idle time = 27.5 Gbit.
80% of the bandwidth is wasted daily!
GenICCP GenICCP
100 byte travel time
DS-0 12.5ms
T-1 0.5ms
Gigabit 0.0008ms
Carrier Ethernet – PBB-TE
After exhaustive evaluation we chose Carrier
Ethernet as our next Transport Infrastructure
PBB-TE supports SONET like switch times
PBB-TE deterministic, connection oriented
services
Carrier Grade reliability and availability
Conclusion
Events that show how sync affect protection
were discussed
Power Systems cover large areas requiring
accurate timing to keep the system balanced.
Evolution of com system is coming
Along with it are the timing requirements
© 2014 Microsemi Corporation. COMPANY PROPRIETARY 50
Power Matters.TM
Synchronization and Timing Evolution
Jim Olsen
May 2016
Power Matters.TM 51 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
Frequency, Time and Phase Synchronization
TA=1/fA
TB=1/fB
fA=fB
Frequency Synchronization
A
B t
t
TA=1/fA
TB=1/fB
fA=fB
Phase Synchronization
A
B t
t
01:00:00 TA=1/fA
TB=1/fB
fA=fB
Time Synchronization 01:00:10
01:00:00 01:00:10
A
B
t
t
TDM SONET Cross Connects
Switches Base Stations Gateways
LTE Advanced CDMA
Channel Banks LTE TDD
Base Stations Computers
Network Management
Set Top Boxes SLA Verification
Power Matters.TM 52 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
Clock Distribution Platforms • Referenced to GNSS/GPS or Cesium Atomic Clocks (PRS)
• DS1 output signals
– Switches
– DACS
– Gateways
– GPON
– SONET ADM
– ROADAMs
• Composite Clock output signals (phase synchronization)
• 64 KHz with built in bipolar violation every 8th bit
– Channel Banks (Multiplexing and cross connection of 56k circuits)
– STPs for SS7 (Mission Critical Application)
Frequency Distribution in the Central Office BITS (Building Integrated Timing Supply)
Synchronous Ethernet (SyncE) uses the same model as
Synchronous SONET to distribute Frequency at the physical layer
Power Matters.TM 53 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
Timing Architecture Time Division Multiplexing (TDM) Centric
Central Office
STP SS7
Channel Bank
IPTV STBs
Composite
Clock
OLT
Primary Reference
Source (PRS)
TDM NEs
DS1
Building Integrated
Timing Supply
NTP
ONT/ONU
GPON
T1
3G BTS
T1
ADM
NTP
ADM
ADM
T1
SONET SONET
T1
CPE
Power Matters.TM 54 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
SONET Mapping
Asynchronous
ADM ADM
F2 F3 F4 F5 F2 F3 F4 F5
Low
Speed
Tributaries The source clock frequency of the tributaries is not affected
OCN
OCN
• The majority of SONET/SDH applications use asynchronous mapping of tributaries into the payload envelope. Asynchronous mapping maintains the source clock of the tributaries
• Asynchronous mapping is designed to eliminate slips by using pointer justifications (a byte stuffing technique)
• A frequency offset between the asynchronously mapped tributaries and the local ADM clock introduces pointer justifications on the tributaries
Network synchronization (frequency alignment of the switching and
transport network) minimizes slips in asynchronously mapped applications
by mitigating jitter and wander on the tributaries
Power Matters.TM 55 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
SONET Mapping
Synchronous
ADM ADM
F2 F3 F4 F5 F2 F3 F4 F5
Low
Speed
Tributaries The source clock frequency of the
SONET/SDH network is imposed on the tributaries
OCN
OCN
• Some of SONET/SDH applications use synchronous mapping of tributaries into the payload envelope. Synchronous mapping imposes the SONET network clock on the tributaries bring mapped
• A frequency offset between the synchronously mapped tributaries and the local ADM clock introduces slips on the tributaries
Network synchronization (frequency alignment of the switching and
transport network) eliminates slips in synchronously mapped applications
Power Matters.TM 56 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
Telecom Profiles
• PTP (Precision Time Protocol) (IEEE 1588)
• GMC is a PTP (1588) server
Frequency Profiles
• Solves Frequency reconstruction issues related to PDV
• Telecom 2008, Layer 3 Unicast
• G.8265.1, Layer 3 Unicast
Phase Profiles
• Solves time transfer errors related to Asymmetry
• G.8275.1, Layer 2 Multicast (Full on Path Support)
• G.8275.2, Layer 3 Unicast (Partial On Path Support)
Timing for Telecom
• What is a PRTC (Primary Reference Time Clock) ? 100 Nanoseconds
• What is a ePRTC (enhanced Primary Reference Time Clock)? 30 Nanoseconds
From Frequency to Frequency, Time and Phase Grand Master Clocks and PTP Profiles
Power Matters.TM 57 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
Systems throughout the world are heavily reliant upon GPS
• Homeland security identified GPS timing as essential to 11 of 18 Critical Infrastructures and Key Resources (CIKR) in the U.S.
• DoD systems have similar issues with GPS dependency
Fundamental issue is that GPS is treated as a “trusted” source of Position, Navigation, and Timing (PNT)
• Few systems have mechanisms to ensure the integrity of GPS information they receive
• GPS devices have essentially achieved insider status within the systems they serve
Ubiquitous nature of GPS and its “insider status” as a source of PNT information make it a significant threat vector for adversaries to exploit
• Vulnerabilities of GPS have been well publicized and are relatively simple attacks
• Impact of a GPS attack is high because many systems cease working or perform poorly when GPS is unavailable
GPS Dependency
Power Matters.TM 58 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
GPS attacks are categorized according to the failure mode they induce
• GPS Jamming: Partial or complete loss of GPS signal
– Most commonly the result of unintentional interference from nearby RF sources
– More complex jamming attacks can be orchestrated by adversaries to make it more difficult to detect the source of the jamming but the result is the same – The GPS receiver fails to receive the GPS signal
• GPS Spoofing: Reception of illegitimate GPS signals
– GPS receiver is “tricked” into tracking GPS-like signals so that it continues to operate but the solution for position and time given by the receiver will be wrong
– These are almost always intentional attacks and can be difficult to detect
Other attacks on GPS such as attacking the GPS control segment or accessing a GPS receiver via its communications ports are classified as cyber attacks
Cesium Atomic Clocks can be deployed to hold frequency and phase in many applications and architectures when GPS becomes unavailable
Characterizing GPS Threats
Power Matters.TM 59 © 2014 Microsemi Corporation. COMPANY PROPRIETARY
Timing for Telecom
• How do you meet and measure stringent time and phase requirements?
• What is a PRTC (Primary Reference Time Clock) ? 100 Nanoseconds
• What is a ePRTC (enhanced Primary Reference Time Clock)? 30 Nanoseconds
Timing for Power Utilities
• PTP Power Profile C.37.238
• What is a Transparent Clock?
The future of Packet Based Timing New PTP Profiles? • Broadcast Video, Financial Trading, IOT
GPS Vulnerability
• Do you need a backup strategy to GPS/GNSS?
• What role do Cesium Atomic Clocks play in addition to GPS/GNSS?
Join us at the WSTS conference for answers to these questions and many more!
Telecom Timing Tools and Technologies are Being Adopted by Other Industries
© 2014 Microsemi Corporation. COMPANY PROPRIETARY
Thank You
ATIS Board of Directors’ Meeting October 20, 2011
Why Industry Needs Time Webinar Wednesday, May 4, 2016
61
www.atis.org/wsts
QUESTIONS?
ATIS Board of Directors’ Meeting October 20, 2011
Why Industry Needs Time Webinar Wednesday, May 4, 2016
Learn More About Sync-Related
Issues at WSTS 2016
62
www.atis.org/wsts
ATIS Board of Directors’ Meeting October 20, 2011
Why Industry Needs Time Webinar Wednesday, May 4, 2016
Thank you for attending
Why Industry Needs Time
All registered attendees will receive a follow up email
containing links to a recording and the slides from this
presentation.
For information on upcoming ATIS events, visit
www.atis.org/events
63
