22
BigFoot project GRIDPOCKET 2012 1 GRID POCKET PERSONAL SMARTGRID SOLUTIONS Prof. Pietro Michiardi (EURECOM) and Filip GLUSZAK (GRIDPOCKET) Sophia Antipolis 20/03/2012
BigFoot project
GRIDPOCKET 2012 1
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Prof. Pietro Michiardi (EURECOM) and Filip GLUSZAK (GRIDPOCKET)
Sophia Antipolis 20/03/2012
Contents SmartGrid applications for BigData
Grid monitoring Smart metering Intelligent home
BigFoot research project Consortium Platform Objectives
Conclusions
GRIDPOCKET 2009 CONFIDENTIAL 2
3
« kWh » to « Energy Services » paradigm shift
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
New needs and opportunities:
Data Services for Energy Environmental
constraints
Market deregulation
Technology disruption
GRIDPOCKET 2012
Smart Grid data analytics market will generate $11.3 billion cumulative from
2011 to 2014 (Pike Research 2011)
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Data sources in SmartGrid
Grid monitoring
Smart Meters Intelligent home / M2M
4 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
5 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Synchrophasors data Phaser Measurement Units provide
phaser information (magnitude and phase angle) in real time
Reporting rate of up to 60 frames per second to Phaser Data Concetrators
6 Source : Siemens.com
Source : Siemens.com
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Example projects Tennessee Valley Authority
(TVA) Hadoop OpenPDC (data from Phaser
Measurement Units) standard project done by Claudera
Data volumes : Serving 9 million consumers Planning 1000 PMDs 30 readings per second 4.2 billion samples per day Need 500TB storage
Application MapReduce oscillation scan :
detection of power grid anomalies, creating power grid maps and evaluating power consumption history
7
Source : claudera.com GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
8 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
9
Smart metering deployments
Electricity, Gas and Water markets 270 millions smart meters in 2013 (ABI, Gartner)
$200 billion investment by 2015 (Pike)
Smart metering projects world-wide (Engage Consulting)
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Volumes of data For 35 million meters (France), data
reading every 10,30, or 60 minutes Generates : 40-200 TB per year (depending
on reading frequency and format) Gas and water grid under deployment
10 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Source : ERDF
!Source : ERDF
Example projects with metering data Ontario Power Authority (OPA) experimentation planned 4.5
million meter collecting data every hour – system wide analysis
Southern California Edison with Teradata – 100TB warehousing for user metering data
Itron partners with IBM, SAP and Teradata to form Active Smart Grid Analytics – enabling applications such as Settlements, Network Loss, and Revenue Protection
Opower - 56 billions meter reads per year, planning to use Hadoop
11 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
12 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Intelligent home applications
13
• Security (detec,on, camera)
• Smart metering
• Ligh,ng control • Smart appliances
• Comfort control / HVAC
• Mul,media
• And more…
GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Volumes of data Multiple standards for sensors inter-
connectivity ETSI M2M framework For 35 million meters,
4-20PB per year in the future (100 sensors per HH)
14 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
15 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
BIG Data Analytics of Digital FOOTprints
BigFoot project consortium
Eurecom (France) Project leader Ecole Polytechnique Fedérale de
Lausanne (Switzerland) TU Berlin / Deutsche Telecom Lab
(T-Lab) (Germany) Symantec (Ireland) GridPocket (France)
Funded by EU FP7 Call 8
16 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
BigFoot – Use case SmartGrid data storage and processing
Consumer billing Consumer Dashboard Web applications:
personalized customer reports; Load analysis: mix, detection of anomalies,
consumer behavioral analysis Consumer segmentation: execution of complex
algorithms to classify consumers based on their consumption patterns and produce, for example, personalized contract;
Assets usage analysis: analysis of geographical consumption patterns and design of predictive algorithms to help operators in provisioning of their electric network.
17 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
BigFoot SmartGrid scenario
18 GRIDPOCKET 2012
BigFoot Platform SmartGrid Apps
WP3Batch Analytics and Interactive Query Engines
WP2
WP4
WP5
Applications
Virtualization Layer
Distributed Data Stores
HadoopDistributedFilesystem
InfrastructureVirtualization
High-level queryto
low-level program translation
Optimization Library
Service-orientedQuery Engine
DistributedDatastore
Data Partitioning and Placement
Data Store Support
ServiceDeployment
ServiceOptimization
Data miningAlgorithms
StatisticalPatternsAnalysis
InteractiveQueries
Virtualization Layer
1 2 3
HadoopMapReduce
Compiler
BigF
oot S
oftw
are
Stac
k
R&D: Workload suites High-level languages Batch and interactive engines Distributed Data stores Beyond “best-effort”: Virtualization Deployment Contributions: ASF 2.0 Hadoop OpenStack
Eurecom experimental platforms
First Generation 100 nodes
single ARM v5, 1.2GHz, 512MB RAM, 64GB
Power efficient 5 watts per node
Second Generation 8 High-end servers 160 Cores 512 GB RAM 100 TB disk GB network
20 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
Conclusions Smart Grid and big data
Generates significant amounts of data New needs and applications are coming
Hadoop technology Promising, however still need to gain on maturity Complex setup and maintenance Interoperability Limited skills and support available in Europe
Next steps BigFoot and other BigData projects GridPocket and BigFoot are open for cooperation Looking for comments and suggestions
21 GRIDPOCKET 2012
GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !
22 GRIDPOCKET 2012
