© Hitachi, Ltd. 2015. All rights reserved.
NASPI WG meeting
Data & Network Management Task Team
Efficient PMU Data Analysis through
High Performance Data Management Platform
Hitachi America Ltd.
Big Data Research Laboratory
10/14/2015
Bo Lucy Yang, Jun Yamazaki, Norifumi Nishikawa,
Hsiu-Khuern Tang, Alex Wang, Anshuman Sahu
© Hitachi, Ltd. 2015. All rights reserved.
1. Platform Architecture
2. High Performance Data Management
3. Integrated PMU applications
Contents
1
4. Visualization
© Hitachi, Ltd. 2015. All rights reserved.
Background
• PMUs have become increasingly popular in North America
• Many PMU data analysis tools have been developed
• Grid dynamics monitoring
• Oscillation detection
• Model validation
Challenges
Increasing PMU data size requires;
• Fast loading of historical data (Hundreds of TB/year)
• Efficient data cleansing against missing and bad data
• Effective data analysis
Motivation
2
To accelerate integrated online/offline analysis
fully utilizing every PMU data into grid operation
PMU installation in North America
0
500
1000
1500
2000
2500
2010 2015
Num
be
r
166
~2,000*1
*1: Silverstein, A. May 2015. NERC Board of Trustees
Meeting, May 5, 2015, "An Update on the North
American SynchroPhasor Initiative"
© Hitachi, Ltd. 2015. All rights reserved.
High performance data management platform for PMU data apps
• High speed database engine
• Integrated PMU applications
• Visualizations
1-1. Platform Architecture
3
Hitachi Visualization
Oscillation
Analysis
Hitachi Data
Analysis
Package
Hitachi Machine
Learning
Hitachi Database
Common Framework
For Third Party Tools
Contingency
Analysis
Other
Tools
© Hitachi, Ltd. 2015. All rights reserved.
1-2. Platform Architecture
4
• Total integration from fast data acquisition to various applications
and effective visualization
Online
PMU
ED
OD
PAA
Historical
PMU
VSA
ED
OD
PAA
VSA
ML
DSA
Case.1
Case.2
…
Hitachi Visualization
Oscillation
Analysis
Hitachi Data
Analysis
Package
Hitachi
Machine
Learning
Hitachi Database
Common Framework For
Third Party Tools
Contingency
Analysis Dynamic Model
Online
PMU SCADA Topology
SE
DSA
MV
Historical
PMU
CA
© Hitachi, Ltd. 2015. All rights reserved.
5×108
4×108
3×108
2×108
1×108A
ddre
ss (
sect
or)
0 1000 2000
Processing time (sec)
2-1. Hitachi database technology
5
Application of the outcome of “Development of the fastest database engine for the era of very large database, and Experiment and evaluation of strategic social services enabled by the database engine” project (Principle
Investigator: Prof. Masaru Kitsuregawa, University of Tokyo and also Director-General, National Institute of Informatics), supported by the Japanese Cabinet Office’s FIRST Program (Funding Program for World-Leading
Innovative R&D on Science and Technology).
*1 A new principle invented by Professor Kitsuregawa and Project Associate Professor Goda (The University of Tokyo).
■ Fully utilizes the hardware (server, storage) resources
■ SQL processing for DB search is automatically divided and executed with a high degree of parallelism
Task Allocation Search Processing I/O Wait Disk I/O
Server
Storage
[Conventional Approach] Sequential Execution
[New Approach] Out-of-Order Execution Principle*1
Search Performance(μs)
Periodic I/O Processing(ms)
【Storage Access Trace in Conventional Approach】
Server
Storage
more than thousands
of tasks
Reduce the processing time dramatically
Fully utilize storage’s performance
0 1000 2000
Processing time (sec)
5×108
4×108
3×108
2×108
1×108Add
ress
(se
ctor
)
【Storage Access Trace in New Approach】
© Hitachi, Ltd. 2015. All rights reserved.
2-2. Performance evaluation
6
110.0
134.3
2.0 3.7
0
20
40
60
80
100
120
140
Query #1 Query #2
Conv. DB
HADB
x 55 x 36
HADB: Hitachi Advanced Database
Query #1
Time series Trend Search
10 minutes, 4 PMU
Query #2
Snapshot Trend Search
5 snapshots, 500 PMU
© Hitachi, Ltd. 2015. All rights reserved.
3-1. Integrated PMU applications
7
Unsupervised data cleansing
• Outlier detection without domain knowledge
• Noise reduction based on data correlation
• Unsupervised, scalable solution
© Hitachi, Ltd. 2015. All rights reserved.
3-2. Integrated PMU applications
8
Automatic abnormality event detection
• Online detection of grid events with measurement-based method
• Historical data mining for similar events
• Updating detection rules from stored historical event data
Start
End
Compute Feature
Vector for PMUs
Event Decision
Update thresholds
• FFT amplitude
• Residue from fitted
complex exponentials
• Spectral density
• Max-min change
• …
© Hitachi, Ltd. 2015. All rights reserved.
4. Visualization
9
Visualization of various applications
• Coordination of information derived from heterogeneous data
• Intuitive display and operator support
© Hitachi, Ltd. 2015. All rights reserved.
Conclusion
10
Conclusion
To utilize both online and historical PMU data for operation…
• Fast DB for acceleration and better efficiency of analysis tasks
• Coordination of various power apps for comprehensive analysis
• Visualization for intuitive awareness with heterogeneous information
Future Plan
• Performance evaluation tests are ongoing on every layers; Database, Applications, and Visualization
• Integration of third party analysis tools to be tested
• Evaluation with real grid data will be planned
© Hitachi, Ltd. 2015. All rights reserved.
Efficient PMU Data Analysis through
High Performance Data Management Platform
10/14/2015
Bo Lucy Yang, Jun Yamazaki, Norifumi Nishikawa,
Hsiu-Khuern Tang, Alex Wang, Anshuman Sahu
END
Hitachi America Ltd.
Big Data Research Laboratory
11