S.2.h Meter Data Management Service

WP4 – Integration of SUNSHINE pilot smart urban services

D4.5 Meter Daata Management service #2

CIP-ICT-PSP-2012-6 – 325161

"This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the

Competitiveness and Innovation Framework Programme by the European Community"

(http://ec.europa.eu/ict_psp).

D4.5

Meter Data

Management Service #2

WP 4 – Integration of SUNSHINE pilot smart urban services

Task 4.2 – Meter data management service

Revision: v1.0 Authors: Stefano Pezzi (SGIS) Luca Giovannini (SGIS)

http://ec.europa.eu/ict_psp



CIP-ICT-PSP-2012-6 – 325161




Dissemination level PU (public)

Contributor(s) Stefano Pezzi, Martina Forconi, Oscar Benedetti, Enrico De Guidi, Luca Giovannini (SGIS), Luis Torres (Meteogrid)

Reviewer(s) Federico Prandi (Graphitech)

Editor(s) Raffaele De amicis (Graphitech)

Partner in charge(s) SGIS

Due date 31/01/2015

Submission Date 31/03/2015

REVISION HISTORY AND STATEMENT OF ORIGINALITY

Revision Date Author Organisation Description

0.1 08/02/2015 Luca Giovannini SGIS Document creation

0.5 19/03/2015 Luis Torres Meteogrid Document review

1.0 20/03/2015 Luca Giovannini SGIS Document finalization

Statement of originality: This deliverable contains original unpublished work except where clearly indicated otherwise.

Acknowledgement of previously published material and of the work of others has been made

through appropriate citation, quotation or both.

Moreover, this deliverable reflects only the author’s views. The European Community is not

liable for any use that might be made of the information contained herein.




CIP-ICT-PSP-2012-6 – 325161




Table of content

Table of content ........................................................................................................................... 3

Acronyms ...................................................................................................................................... 3

1 Introduction ....................................................................................................................... 5

2 Meter data management service....................................................................................... 5

2.1 Overall architecture ....................................................................................................... 6

3 Hardware components ...................................................................................................... 9

3.1 Main platform servers ................................................................................................. 10

3.1.1 Backend server ...................................................................................................... 10

3.1.2 FTP server ............................................................................................................. 10

3.1.3 Webserver server .................................................................................................. 10

3.1.4 Firewall server ....................................................................................................... 11

4 Software components ...................................................................................................... 12

4.1 Sensor DB (52° North SOS 4.0) ..................................................................................... 12

4.1.1 Database structure ............................................................................................... 13

4.1.2 Database management ......................................................................................... 16

4.2 FTP ingestion ................................................................................................................ 18

4.3 Meter data processing ................................................................................................. 19

4.4 CSV loader .................................................................................................................... 20

4.5 Weather ingestion ....................................................................................................... 21

4.6 GreenButton ingestion ................................................................................................ 23

4.7 Sensor data access ....................................................................................................... 25

5 Sunshine Weather Platform ............................................................................................. 27

5.1 Platform architecture .................................................................................................. 27

5.1.1 Weather Data Collecting Service .......................................................................... 28

5.1.2 Weather Data Exhibition Service - WMS – WFS ................................................... 28

5.2 Software components .................................................................................................. 29

5.2.1 Core modules ........................................................................................................ 29

5.2.2 Zope Object Database ........................................................................................... 30

5.2.3 Data ingestion in Sigym3 ...................................................................................... 31

5.2.4 Data Access ........................................................................................................... 31

Annex I - Sensor DB schema (52° North SOS 4.0)....................................................................... 32

Annex II – Mapping weather WFS to Sensor DB ........................................................................ 33

Acronyms




CIP-ICT-PSP-2012-6 – 325161




CPU Central Processor Unit

CSV Comma Separated Values

DB Data Base

DMZ DeMilitarized Zone

FTP File Transfer Protocol

IT Information Technology

LAN Local Area Network

SOS Sensor Observation Service




CIP-ICT-PSP-2012-6 – 325161




1 Introduction

This document is an annex to the final deliverable of the Meter Data Management Service of

the Sunshine project. The following chapters describe the implementation of the service with

respect to hardware components, interaction with pilot infrastructures and functional

software modules. The reader can also refer to deliverable D1.5 “SUNSHINE System

architecture – final” for an overview of this service and of its relation with the other elements

of the Sunshine platform.

2 Meter data management service

During the development of SUNSHINE’s platform the scope of Meter Data Management

Service has been extended from the management of meter data to the management of data

coming more generally from sensors. The reason being that SUNSHINE’s pilots are providing

not just consumption data from meters, but also several other types of data from other

sensors that also need to be collected and managed to support SUNSHINE’s services and

piloting activities. More specifically, the Meter Data Management service manages:

Consumption data from energy meters in pilot buildings;

Indoor temperature readings from indoor sensors in pilot buildings;

Weather observation and forecasts data from a dedicated weather service from

Meteogrid;

Status data (dimming level, hours of use, etc) from pilot lamps and light-lines.

All these data types are stored in the same Sensor DB and managed with common

procedures, which will be described in this deliverable. In order to avoid redundancies, the

detailed description of the management of status data for lamps and light-lines, which is also

part of the Remote System Management service, has not been repeated here and can be

found in deliverable D4.6 “Remote System Management Service n2”.

In brief, the purpose of the Meter Data Management Service is to:

Gather all the data coming from meters/sensors;

Elaborate them and generate derived data, if necessary (Section 4.6);

Store them in a normalized format into a repository (Section 4.1) from which they can

be retrieved through a standard interface (Section 4.7).

For consumption data from meters two data flows are supported by the platform, in order to

enable both the pilots with low IT capacities and those who are able to develop software

components:




CIP-ICT-PSP-2012-6 – 325161




Simple FTP transfer of CSV file;

Feed from Green Button web service.

Two service endpoints correspond to these types of flows:

FTP Ingestion service (Section 4.2);

Green Button Ingestion service (Section 4.6).

To maximize the reuse of the developed software, the Green Button Ingestion service doesn’t

load directly the data in the DB, but produces a CSV file that is treated by a dedicated

component, the CSV loader (Section 4.4), that is the same used by the FTP Ingestion service.

The flow of indoor temperature readings is channelled through the Green Button web service

(Section 4.6) as all the pilots actually providing such data flow have implemented the needed

infrastructure.

The flow of weather data (Section 4.5) is established via a scheduling procedure that is

planned to pull data daily from the dedicated Web Feature Services (WFS) set up by partner

Meteogrid (whose dedicated platform is described in detail in Chapter 5).

The flow of lamp status data is managed by the Remote System Management service (see

deliverable D4.6 “Remote System Management Service n2”) that ensures the uniform

ingestion of such data via the ingestion API of the Sensor DB.

2.1 Overall architecture

The overall architecture of Sunshine is shown in Figure 1, we’ll now roughly describe the

whole system and then we’ll focus on the components involved in the meter data

management service. The diagram is quite complex and the different colours differentiate the

various functional sections of the entire system:

Red modules compose the security layer;

Green modules compose the meter data management service, overseeing the

ingestion of different data flows into Sunshine’s sensor database;

Blue modules compose the remote system management service, managing light line

control and monitoring;

Grey modules compose the building pre-certification service, managing building

geodata and estimating building heath needs.

Services are implemented by the boxes representing functional units of software and by the

blue circles that define the interfaces.

In the left side of the diagram there are some of the nodes that deliver information to the

Sunshine’s platform plus the Sunshine’s client application. An exception is represented by the




CIP-ICT-PSP-2012-6 – 325161




brown node on the right side, at the bottom, that is a virtual machine with City of Ferrara’s

software components that is hosted at Sinergis’ premises1.

The server components are on the right portion of the diagram: here the red ones represent

the security layer that will be described more in detail in the D4.6 Annex on Security Layer.

The layer doesn’t add any additional functionality, but intercepts the requests that are sent to

services that need to be protected. Even the reverse proxies defined inside the web server

are in red, because this way to expose services on the internet adds also a little degree of

protection besides other advantages.

1 Physically the VM is located in the same LAN of the Sunshine’s platform, but the interactions with its

components are made via web protocols so it can be deployed to the pilot’s premises without any

problem.




CIP-ICT-PSP-2012-6 – 325161




Figure 1 - Overall deployment diagram



D4.5 Meter Data Management service #2

CIP-ICT-PSP-2012-6 – 325161




3 Hardware components

Figure 2 - Physical and virtual server

The SUNSHINE specific servers are deployed on a virtualized IT infrastructure based on

VMware vSphere software (the “Back end VM Server” in Figure 2). The underlying hardware

has the following features:

CPU Architecture x86_64

CPU Number 2 (6 core each)

CPU Clock 2300 MHz

Hard Drive Space 12TB

As SGIS plays in SUNSHINE also the role of technical partner of the Ferrara pilot, the same

infrastructure hosts as well the pilot’s server (sunshine-fe.nco.inet) used to collect the meter

data from the various buildings and to transmit them to the SUNSHINE platform. The Ferrara’s

server shares only the virtualization infrastructure and the internet domain (sinergis.it) with

the central platform, but all the communications between the SUNSHINE platform and the

pilot’s server passes through internet protocols.

Other servers that are functional to the SUNSHINE platform, but play generic roles (firewall,

web server, ftp server) are based on separate hardware in order to achieve a correct and

secure network topology and to fit in the existent SGIS internet infrastructure. In particular we




CIP-ICT-PSP-2012-6 – 325161




have the FTP server (ftp-vm) that is a VM hosted in another virtualization infrastructure

located in the DMZ. Then we have the web server (coreweb2) that is a physical server located

in the DMZ and finally the firewall. More details are given in the following paragraphs.

In the diagram of Figure 2 also a sunshine-revdb.nco.inet virtual server is shown, but its role is

descripted in D4.3.

3.1 Main platform servers

The main components are deployed on the backend server that is positioned in the internal

LAN in SGIS’ premises in Casalecchio di Reno, Bologna; the others server involved are:

the FTP server, that is located in a separate VM server positioned in the DMZ for

security reasons;

the web server is also a physically separated server; this server is used by SGIS

company not only for the SUNSHINE project, but for all the other company’s project

and customers.

the firewall; also this server is a company’s asset that is functional to all projects and

that is used to guarantee security for the server that are exposed on the internet.

3.1.1 Backend server

Operating System Ubuntu 12.04.3 LTS 64bit

Host Name

IP address

3.1.2 FTP server

Operating System Ubuntu 12.04.3 LTS 64bit

Host Name

IP address

DNS name

3.1.3 Webserver server

Operating System CentOS 6.4 64bit

Host Name

IP address




CIP-ICT-PSP-2012-6 – 325161




3.1.4 Firewall server

Operating System Slackware 12.0 64bit

Host Name

IP address




CIP-ICT-PSP-2012-6 – 325161




4 Software components

The components are described in details in the sections of this chapter and their mutual

relation is described in the figure below (in green).

Figure 3 - The meter Data Management service components

4.1 Sensor DB (52° North SOS 4.0)

It is the final repository of all sensor data. A database implementation of OGC Sensor

Observation Service2 (SOS) standard protocol developed by 52° North3 has been installed and

configured on Sunshine’s platform for this purpose. It is implemented as a PostgreSQL DB and

also makes use of the spatial extension PostGIS.

2 http://www.opengeospatial.org/standards/sos 3 http://52north.org/communities/sensorweb/sos/index.html


http://www.opengeospatial.org/standards/sos

http://52north.org/communities/sensorweb/sos/index.html



CIP-ICT-PSP-2012-6 – 325161




4.1.1 Database structure

Annex I of this document reports the UML schema of the sensor DB. The main constructs of

the DB schema are the following:

Codespace: univocally identifies the pilot;

Feature of Interest: represents the geo-object who is measured by sensors. The FOI is

the means of locating (geocoding) the measuring points via their coordinates;

Observable Property: identifies the property observed by the sensor (e.g. gas

consumption, temperature or lamp dimming level);

Procedure: identifies a physical /virtual sensor which produces the observations

associated with the offerings;

Offering: a logical grouping of observations that are related to each other, which are

jointly offered by a service.

The following sub-sections describe the uniform and consistent naming convention that has

been defined for these constructs.

Codespace

CodeSpaceUri (examples) CodeSpace Pilot

http://www.sunshineproject.eu/swe/project-pilot/ROV ROV Rovereto

http://www.sunshineproject.eu/swe/project-pilot/FER FER Ferrara

http://www.sunshineproject.eu/swe/project-pilot/BAS BAS Bassano

http://www.sunshineproject.eu/swe/project-pilot/LAM LAM Lamia

http://www.sunshineproject.eu/swe/project-pilot/PAO PAO Paola

http://www.sunshineproject.eu/swe/project-pilot/TRN TRN Trentino

http://www.sunshineproject.eu/swe/project-pilot/VDN VDN Val di Non

http://www.sunshineproject.eu/swe/project-pilot/HRV HRV Croatia

CodeSpaceUri: <radix>/CodeSpace

Feature of Interest

FeatureofinterestUri (examples)

http://www.sunshineproject.eu/swe/featureOfInterest/FER/building/ScuolaACosta

http://www.sunshineproject.eu/swe/featureOfInterest/FER/building/PalazzinaMarfisaDEste

http://www.sunshineproject.eu/swe/featureOfInterest/HRV/weatherstation/06

http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lamp/101

http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lamp/102

http://www.sunshineproject.eu/swe/featureOfInterest/BAS/lightline/PiazzaGuadagnin

http://www.sunshineproject.eu/swe/featureOfInterest/LAM/building/SchoolTechApplications2

http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lightline/RotondaMarco




CIP-ICT-PSP-2012-6 – 325161




FeatureofinterestUri: <radix>/CodeSpace/<natura_della_Feature>/<Identificativo>

FoI nature

Building

Lightline

Lamp

Weatherstation

Observable property

ObservablepropertiesUri (examples)

http://www.sunshineproject.eu/swe/observableProperty/ELEC

http://www.sunshineproject.eu/swe/observableProperty/TEMP

http://www.sunshineproject.eu/swe/observableProperty/IPOW

http://www.sunshineproject.eu/swe/observableProperty/DIMM

http://www.sunshineproject.eu/swe/observableProperty/LIFE

http://www.sunshineproject.eu/swe/observableProperty/IRRA

http://www.sunshineproject.eu/swe/observableProperty/GASR

ProcedureUri: <radix>/<CodeProperty>

CodeProperty Description

ELER Electrical Energy Consumption, relative

GASR Gaseous Fuel Consumption, relative

THER Thermal Energy Consumption, relative

LIQR Liquid Fuel Consumption, relative

SLDR Solid Fuel Consumption, relative

STMR Steam Fuel consumption, relative

ELEC Electrical Energy Cost

GASC Gaseous Fuel Cost

THEC Thermal Energy Cost

LIQC Liquid Fuel Cost

SLDC Solid Fuel Cost

STMC Steam Fuel Cost

ELEA Electrical Energy Consumption, absolute

GASA Gaseous Fuel Consumption, absolute

THEA Thermal Energy Consumption, absolute

IPOW Lamp instantaneous power

DIMM Lamp dimming

LIFE Lightline usage hours




CIP-ICT-PSP-2012-6 – 325161




ONOF Lamp CB Status

NALL Lightline CB Number

N-OK Lightline CB Accreditate

SUNR Lightline sunrise time

SUNS Lightline sunset time

ICUR Lamp current

COSF Lamp Cos(phi)

IVOL Lamp voltage

SCDL Lamp scheduler

ACNT Lamp accounting

TEMP Temperature

WIND Wind Speed Module

IRRA Short Wave Irradiation

Procedure

ProcedureUri (examples)

http://www.sunshineproject.eu/swe/procedure/ROV-100

http://www.sunshineproject.eu/swe/procedure/VDN-F00

http://www.sunshineproject.eu/swe/procedure/BAS-F00

http://www.sunshineproject.eu/swe/procedure/TRN-M01

http://www.sunshineproject.eu/swe/procedure/TRN-T01

ProcedureUri: <radix>/<codespace>-<alphaNumeric>

alphaNumeric:

Building/shelter consumption data: numeric between 000 and 999

Lamp/line consumption and state data:

o Hundreds identify the line

o Tens and units identify lamps belonging to the same line

Weather Forecast data: F<n> with n ∈ [0..99]

Weather Meteorological conditions: M<n> with n ∈ [0..99]

Indoor Temperature data: T<n> with n ∈ [0..99]

Offering

Offering (examples)

http://www.sunshineproject.eu/swe/offering/ROV-F01_TEMP_CEL_1h

http://www.sunshineproject.eu/swe/offering/LAM-M01_TEMP_CEL_1h

http://www.sunshineproject.eu/swe/offering/HRV-001_ELER_KWH_1h




CIP-ICT-PSP-2012-6 – 325161




http://www.sunshineproject.eu/swe/offering/FER-001_GASR_MCU_1d

OfferingUri: <radix>/CodeSpace-

<alphaNumeric>_<codeProperty>_<uom>_<frequenza_di_campionamento>

Sampling frequency Description

10 Every 10 minutes

15 Every 15 minutes

1h Hourly

1d Daily

1m Monthly

ir Irregular frequency

Unit of measure Description

KWH "kWh"

CEL "°C"

MCU "m3"

EUR "Euro"

WAT "W"

PRC "%"

HRS "hours"

BOO "0/1"

NUM "elements"

HMS "date"

AMP "Ampere

CDL "codelist"

KWA "kW"

KMH "m/s"

WM2 "W/m2"

4.1.2 Database management

The management of the database structure have been performed via:

The use of the SOS APIs for the creation/deletion of new procedures (insertSensor,

deleteSensor);

The use of custom SQL scripts for the creation/deletion of new offerings. Scripts take

care to modify all the entries related to offerings in the various DB tables.

The creation of a new offering implies the following actions on the DB:




CIP-ICT-PSP-2012-6 – 325161




DB TABLE Action

offering

Create entry with identifier, e.g. “LAM-

M01_TEMP_CEL_1h”.

observableproperty

Verify entry existence for “Observable Property” of the

offering, e.g. “TEMP”.

unit

Verify entry existence for "Unit" of the offering, e.g.

“CEL”.

featureofinterest Verify entry existence for “Feature of Interest” of the

offering, specifying:

featureofinteresttypeid among options in table

featureofinteresttype

codespaceid

series

Insert entry with

featureofinterestid

observablepropertyid

procedureid

observationconstellation

Insert entry with

observablepropertyid

procedureid

observationtypeid

offeringid

offeringallowedobservationtype

Insert entry with

offeringid

observationtypeid

offeringallowedfeaturetype

Insert entry with

offeringid

featureofinteresttypeid

relatedfeature

Verify entry existence with

Featureofinterestid

offeringhasrelatedfeature

Insert entry with

relatedfeatureid

offeringid




CIP-ICT-PSP-2012-6 – 325161




DB management view

The DB management view is a fundamental tool for the set-up and maintenance of the DB

structure and for the monitoring of the correct flowing of data into the DB. At runtime the

Sensor DB has hundreds of offerings and a manual check on the correctness of all offerings

would be unfeasible. As Figure 4 shows, the view has one entry per offering and for each entry

it specifies:

Feature of Interest identifier

Procedure identifier

Offering identifier

Observable Property identifier

Unit of measure

Reading Frequency

Observations count for the offering

Timestamp of first observation

Timestamp of last observation

Figure 4 The Db management view

4.2 FTP ingestion

It’s a scheduled program that checks if any CSV file to be loaded into the database is arrived

on the dedicated FTP folders. The files are removed from the folders and passed to the CSV

loader component that does the real job of inserting data in the DB.

Detailed guidelines have been prepared describing how to format name and content of CSV

files in order to allow the automatic ingestion procedure to take place smoothly.

Guidelines are available in the deliverable D2.4 Annex “Specifications_for_data_ingestion_via

sunshine_FTP”.




CIP-ICT-PSP-2012-6 – 325161




In order to configure the service, pilots compiled a mapping file that has the purpose of

describing the basic properties of the meter/sensor and allows to identify which pilot building

or light line does it belong to. An example of the content of a meter mapping file for the pilot

in Lamia is shown in Figure 5.

Figure 5 – Meter mapping file

4.3 Meter data processing

Some elaborations are needed before the real data ingestion. For instance, some aggregations

are pre-calculated in order to avoid this task to the client. Another kind of processing is the

interpolation for those readings that have been collected at irregular intervals of time (some

kinds of data flows have this characteristic).

This module is specific to the consumption data from sensor reading and its role is that of

guaranteeing a uniform format for its storage and therefore a cleaner server/client

interaction. Stored consumption data has the following format:

Consumption is relative to the last reading interval;

Readings have a regular frequency;

Available frequencies are: every 15 minutes (10 minutes for TNET), hourly and daily.

These constraints do not apply to consumption data coming from billing documentation that

are stored in the BD as is.

In order to guarantee the above-mentioned format, three main processing tasks that have

been developed:

Data resampler, from irregular frequency to regular frequency;

Differential consumption, from absolute readings to relative readings (keeping track

of the last known absolute reading);

Data aggregator, from high frequency regular relative readings to low frequency

regular relative readings (e.g. from 15 minutes frequency to hourly and daily).




CIP-ICT-PSP-2012-6 – 325161




4.4 CSV loader

This component accesses the DB for inserting the readings in the proper tables. Before doing

that, if it has been configured to do so, the files are pre-processed by the “Meter data

processing” component.

Readings are inserted in the DB using SOS insertion API, insertObservation.

What follows is an example of an insertObservation call: <?xml version="1.0" encoding="UTF-8"?>

<env:Envelope

xmlns:env="http://www.w3.org/2003/05/soap-envelope"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://www.w3.org/2003/05/soap-envelope

http://www.w3.org/2003/05/soap-envelope/soap-envelope.xsd">

<env:Body>

<sos:InsertObservation service="SOS" version="2.0.0"

xmlns:sos="http://www.opengis.net/sos/2.0"

xmlns:swes="http://www.opengis.net/swes/2.0"

xmlns:swe="http://www.opengis.net/swe/2.0"

xmlns:sml="http://www.opengis.net/sensorML/1.0.1"

xmlns:gml="http://www.opengis.net/gml/3.2"

xmlns:xlink="http://www.w3.org/1999/xlink"

xmlns:om="http://www.opengis.net/om/2.0"

xmlns:sams="http://www.opengis.net/samplingSpatial/2.0"

xmlns:sf="http://www.opengis.net/sampling/2.0"

xmlns:xs="http://www.w3.org/2001/XMLSchema"

xsi:schemaLocation="http://www.opengis.net/sos/2.0

http://schemas.opengis.net/sos/2.0/sos.xsd

http://www.opengis.net/samplingSpatial/2.0

http://schemas.opengis.net/samplingSpatial/2.0/spatialSamplingFeature.xsd">

<sos:offering>http://www.sunshineproject.eu/swe/offering/ROV-

213_ICUR_AMP_ir</sos:offering>

<sos:observation>

<om:OM_Observation gml:id="o1">

<gml:identifier

codeSpace="http://www.sunshineproject.eu/swe/project-

pilot/ROV">http://www.sunshineproject.eu/swe/observation/ROV-213_ICUR_AMP_ir/2015-03-11

11:53:24</gml:identifier>

<om:type

xlink:href="http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_Measurement"/>

<om:phenomenonTime>

<gml:TimePeriod gml:id="Ki.ObsTime.1">

<gml:beginPosition>2015-03-

11T04:50:47.000+01:00</gml:beginPosition>

<gml:endPosition>2015-03-11T05:50:47.000+01:00</gml:endPosition>

</gml:TimePeriod>

</om:phenomenonTime>

<om:resultTime>

<gml:TimeInstant gml:id="Ki.resTime.1">

<gml:timePosition>2015-03-

11T05:50:47.000+01:00</gml:timePosition>

</gml:TimeInstant>

</om:resultTime>

<om:procedure

xlink:href="http://www.sunshineproject.eu/swe/procedure/ROV-213"/>

<om:observedProperty

xlink:href="http://www.sunshineproject.eu/swe/observableProperty/ICUR"/>

<om:featureOfInterest

xlink:href="http://www.sunshineproject.eu/swe/featureOfInterest/ROV/lamp/213"/>




CIP-ICT-PSP-2012-6 – 325161




<om:result xsi:type="gml:MeasureType" uom="A">29</om:result>

</om:OM_Observation>

</sos:observation>

</sos:InsertObservation>

</env:Body>

</env:Envelope>

4.5 Weather Data ingestion

Weather data are served by two WFS services set up by partner Meteogrid from a

dedicated platform described in detail in Chapter 5. The ingestion process queries the

WFS services, collects the data for each meteorological quantity (temperature,

irradiation, wind speed), and loads it into the Sensor DB. More in details here are the

WFS services:

WFS forecast

A three-day forecast with hourly resolution is available daily for each pilot’s location.

WFS observation

Measured or interpolated weather conditions are available with hourly resolution for

each pilot’s location.

From both WFSs the feature types that are read and ingested to the SOS are:

1. temperature_s: temperature in °C (Celsius Degrees)

2. wind_speed_s: wind module in m/s

3. short_wave_irradiation_s: short wave irradiation in W/m2

Each feature type has the following properties, as can be seen in Figure 6:

msGeometry (gml:Point)

name: name of observatory (example: ZAGREB165)

timestamp (data are available with hourly frequency)

value

type

observatory_id

<gml:featureMember>

<sigym:temperature_s fid="temperatures_2015-03-13

01:00:00+01:00,temperatura_obs_int,165">

<sigym:msGeometry>

<gml:Point srsName="EPSG:4326">

<gml:coordinates>15.965225,45.807468999999998</gml:coordinates>

</gml:Point>

</sigym:msGeometry>

<sigym:name>ZAGREB165</sigym:name>

<sigym:timestamp>2015-03-13T01:00:00+01:00</sigym:timestamp>

<sigym:value>6.0</sigym:value>

<sigym:mode/>

<sigym:type>temperatura_obs_int</sigym:type>




CIP-ICT-PSP-2012-6 – 325161




<sigym:observatory_id>165</sigym:observatory_id>

</sigym:temperature_s>

</gml:featureMember>

Figure 6 - Feature example of temperature_s from observation WFS

Each feature type of both WFS (forecast and observation) has 83 feature members

representing the observatories. The observatory name and id are the same in both WFSs.

Annex II describes the mapping between feature members of the WFS with procedures of the

SOS.

The mapping between WFS and SOS has the following main characteristics:

a feature member of the WFS (observatory) correspond in the SOS to:

o one FeatureOfInterest;

o two Procedures (one for the forecast and one for the observation);

o six Offerings; three offering (temperature, wind, irradiation) for forecast and

for observation.

a value of the WFS correspond to an Observation in the SOS

The Procedure name in the SOS is: <pilot>-F/M<nn> where F=forecast and M=measure

(observation)

The service that reads the WFS and ingest the data to SOS:

is scheduled once a day at 10:10:00 AM

the service makes 3 GetFeature calls to the Forecast WFS and 3 GetFeature calls to

the Observation WFS, one call for each feature type (temperature_s, wind_speed_s,

short_wave_irradiation_s);

the GetFeature call response for the Forecast WFS returns the information for the

next 3 days (72 hours) for the 83 observatories. The service deletes the old forecast

and ingests the values into the SOS.

for the Observation WFS the service calls a GetFeature using the TIME parameter to

specify a range covering one day (from the day before of current date of scheduling

to the current scheduling). The response to the GetFeature call returns the

information for 24 hours for each of the 83 observatories; the values are ingested

into the SOS (and no deletion of old values is done).

The ingestion module refers to a configuration file (importMeteoDataConfiguration.xml) that

allows configuring:

Ingestion scheduling (hour, minute, second)

URL of WFSs

WFS Featuretype names

Mapping between “WFS observatory id” and “SOS procedure code”




CIP-ICT-PSP-2012-6 – 325161




CodeSpace Identifier of the Pilot

FeatureOfInterest Identifier

4.6 GreenButton ingestion

A Smart Energy Grid has many different nodes (generation facility, distribution network, utility

company data storage facility, smart meters, local data concentrator, etc) and actors (energy

producer, energy deliverer, energy seller, energy customer, service provider company, etc)

and consequently many different types of communication channels, each with its own set of

possible communication protocols.

In the context of the Sunshine project, the pilot partner has both the role of energy user and

the role of data storage facility, while the Sunshine platform is the service provider company

that needs access to the consumption data stored in the storage facility in order to provide

value-adding services. This is exactly the use case typology that is covered by the GreenButton

protocol4, the international data exchange standard for consumption data that have been

chosen to be used in SUNSHINE’s platform. The Sunshine platform does not communicate

directly with the smart meters of the pilots and is therefore not implementing any protocol

related to direct communication with smart meters.

Green Button is an initiative of the U.S. Government that aims to allow consumers to access

their own meter data about energy consumption. In the original context, the name Green

Button denotes the whole initiative, but in our project we will call GB only the suite of web

services that makes the data access possible. GreenButton protocol and API were identified as

the best option to implement the use case described above for several reasons:

It is already in operational use;

It is supplied with examples and demos;

It is a standard aiming at becoming international5

So GreenButton provides Sunshine with a protocol already implemented and operational in a

real use case scenario.

This module consumes a REST service that returns an XML that is compliant with the AtomPub

standard that is the underlying standard of the Green Button. Concerning the handling of

authentication and authorization in the data communication process between the pilots and

the Sunshine platform, GreenButton protocol relies on OAuth protocol which has a

widespread use for the use cases we are concerned with (three-partner authorisation

between data owner, data custodian, data user). For the sake of truth, we must highlight that

there is not a real “service endpoint” for this kind of flow. As shown in Figure 7, the meter

data producers actually acts like server, publishing their readings and the SUNSHINE platform

4 http://energy.gov/data/green-button 5 http://www.sgip.org/PAP-20-Green-Button-ESPI-Evolution


http://energy.gov/data/green-button

http://www.sgip.org/PAP-20-Green-Button-ESPI-Evolution



CIP-ICT-PSP-2012-6 – 325161




acts like a consumer of the feeds. The roles are therefore switched and the GreenButton

ingestion service must be configured with the address of the pilot’s feed to read.

Detailed guidelines have been prepared describing the GreenButton standard, its security and

authentication module based on OAuth protocol6 and providing instruction on how to

implement a web service supporting the protocol. Guidelines are available in deliverable

annex “D4.5 Annex - Specifications_for_data_ingestion_via_GreenButton_v1.2”.

Figure 7 - Consuming a GreenButton service

Support was given via Skype telcos and mail exchange to pilots that decided to implement a

web service based on GreenButton protocol to expose consumption data. The most relevant

issues were related to the securization of the infrastructure:

exchange of OAuth security token between pilot and central infrastructure;

activation of HTTPS secure protocol;

6 http://oauth.net/


http://oauth.net/



CIP-ICT-PSP-2012-6 – 325161




Moreover, a mapping was established for each pilot between the data structure of the

GreenButton DB residing in the pilot infrastructure and the data structure of the central

Sensor DB. This mapping was requested in the guidelines disseminated to pilots, where also a

template for the mapping itself was provided. Figure 8 shows a compiled example of such

mapping table.

Figure 8 – Mapping between GreenButton DB structures (usage point, meter reading,

reading type) and Sensor DB (meter identifier, last column).

4.7 Sensor data access

The last module, in logical order, is the sensor data access. Access to sensor data is

guaranteed by two services:

Web feature service (see Figure 9):

o To publish the identifiers of the different sensors and data flows via the

linking with their geographical position.

Sensor observation service APIs:

o GetCapabilities: returns an XML service description with information about

the interface (offered operations and endpoints) as well as the available

sensor data, such as the period for which sensor data is available, sensors that

produce the measured values, or phenomena that are observed;




CIP-ICT-PSP-2012-6 – 325161




o DescribeSensor: provides sensor metadata in SensorML. The sensor

description can contain information about the sensor in general, the identifier

and classification, position and observed phenomena, but also details such as

calibration data;

o GetObservation: allows pull-based querying of observed values, including

their metadata. The measured values and their metadata is returned in the

Observations and Measurements format (O & M);

o GetResult: provides the ability to query for sensor readings without the

metadata given consistent metadata (e.g. sensor, observed object).

Figure 9 – Sensor locations (blue dots) and identifiers (attributes in the table)

exposed via a Web Feature Service.




CIP-ICT-PSP-2012-6 – 325161




5 Sunshine Weather Platform

This chapter describes the platform implemented at Meteogrid premises that is responsible

for the provision of weather data, both observation and localized forecasts, to the central

SUSNHINE’s platform for all pilot locations.

5.1 Platform architecture

In order to support SUNSHINE’s platform services Meteogrid’s team has designed and set up

an IT platform that can generate specific weather forecasts for the SUNSHINE pilot areas,

collect information from NOAA, ECMWF and local weather centres and compute a specific

forecast downscaling for each building.

This platform consists of two dedicated servers

MeteoGrid Weather Forecast server;

Sigym3 MeteoServer.

MeteoGrid Weather Forecast server houses a WRF implementation. WRF - Weather Research

and Forecasting model is a numerical weather prediction system customized in much of the

public and private meteorological services worldwide.

Below major Sunshine customization implementation details are offered:

Latitudes: 34.77º - 51.59º with resolution of 11.7 km.

Longitudes: 3.12º - 30.08º with resolution of 11.7 km

Temporal resolution output: 1 hour

Temporal scope: 72 hours

Output variables: T2, Q2, PSFC, ACSWDNB, ACLWDNB, RAINNC, rainc, U10, V10,

CLDFRA, CLT, CLL

settings:

- Micro-physics: WRF Single-moment three-class and five-class Schemes; Hong, Song-

You, Jimy Dudhia, and Shu-Hua Chen, 2004: A revised approach to ice microphysical

Processes for the bulk parameterization of clouds and precipitation. Mon. Wea. Rev.,

132, 103-120.

- Cumulus parameterization: Kain-Fritsch Scheme; Kain, John S., 2004: The Kain-Fritsch

convective parameterization: An update. J. Appl. Meteor., 43, 170-181.

- Shortwave and Longwave: Shortwave and Longwave CAM Schemes; Collins, William

D., et al., 2004: Description of the NCAR Community Atmosphere Model (CAM 3.0).

NCAR Tech. Note NCAR / TN-464 + STR. 214 pp.

Urban Surface: Urban Canopy Model




CIP-ICT-PSP-2012-6 – 325161




Sigym3 MeteoServer integrates a reinterpretation service of weather forecasting based on

precision procedures and statistical downscaling algorithms; a set of algorithms and models

for the generation of derived variables of interest; and a GIS platform that integrates

meteorological data, derived variables and other spatial data, considering and managing the

time axis.

In addition to the necessary adaptations to meet the needs of the project, specific services for

collection and presentation of the information managed by the system have been developed:

the Weather Data Collecting Service

the Weather Data Exhibition Service

5.1.1 Weather Data Collecting Service

In brief, the purpose of the Weather Data Collecting Service is to:

Gather local observations collected from local weather services or building

meteorological stations;

Gather forecast data from NOAA, ECMWF and MeteoGrid Weather Forecast

dedicated server;

Elaborate them and generate derived data, geo-interpolation processes plus unit

conversion (different weather services use different units for meteorological

variables);

Store them in a normalized format into a repository from which they can be retrieved

through a standard interface (WFS).

The flow of ECMWF and MeteoGrid forecast data are channelled through an FTP Ingestion

service. Data are received twice a day to update forecast information.

The flow of NOAA forecast data and METAR observations data are channelled through the FTP

Ingestion service. Data are gathered via a scheduling procedure that pulls data twice a day to

update forecast information and meteorological observations inside the Weather Platform.

The flow of local weather observation data is managed by the Web Scraper Ingestion and the

FTP Ingestion services that ensure the ingestion of such data every hour.

5.1.2 Weather Data Exhibition Service - WMS – WFS

Collected and processed data is exposed through WMS and WFS services. The data flows

towards Sunshine’s platform via the WFS service. SGIS has set up a scheduled procedure to

pull data daily from the dedicated Web Feature Services (WFS).




CIP-ICT-PSP-2012-6 – 325161




Sunshine-Weather specific servers are deployed on a virtualized IT infrastructure based on

KVM allowing a flexible architecture. The underlying hardware has the following features:

Sigym3 MeteoServer

CPU Architecture x86_64

CPU Number 2 (8 cores)

CPU Clock 2300 MHz

Hard Drive Space 2x2TB (RAID 1)

MeteoGrid Weather Forecast server

CPU Architecture AMD ABU DHABI

CPU Number 2 (16 cores each)

CPU Clock 2300 MHz

Hard Drive Space 8TB

5.2 Software components

All the software used in this platform is free Open-source software (OSS). All the used free

software is widely tested and used in all the software community, throughout the whole

lifecycle of the software projects, it is worldwide tested and it is used in all types of projects,

from small to large scale and, also, from small to large companies.

5.2.1 Core modules

Operating System: Ubuntu (http://www.ubuntu.com/).

Ubuntu is a Debian-based Linux operating system. Ubuntu releases updated versions

predictably – every six months – and each release receives free support for nine months

(eighteen months prior to 13.04) with security fixes, high-impact bug fixes and conservative,

substantially beneficial low-risk bug fixes.

Data Base: PostGIS (http://postgis.net/).

PostGIS is a spatial database extender for PostgreSQL object-relational database. It adds

support for geographic objects allowing location queries to be run in SQL.

Programming Language: Python (https://www.python.org/).

Python is a widely used general-purpose, high-level programming language. Its design

philosophy emphasizes code readability, and its syntax allows programmers to express

concepts in a few lines of code. The language provides constructs intended to enable clear

programs on both a small and large scale.

Apache HTTP Server (https://httpd.apache.org/).


http://www.ubuntu.com/

http://postgis.net/

https://www.python.org/

https://httpd.apache.org/



CIP-ICT-PSP-2012-6 – 325161




Apache HTTP Server provides a secure, efficient and extensible server that provides HTTP

services in sync with the current HTTP standards.

NGINX (http://nginx.org/).

NGINX is an open source reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP

protocols, as well as a load balancer, HTTP cache, and a web server (origin server).

5.2.2 Zope Object Database

Zope Object Database (ZODB) (http://www.zodb.org/en/latest/).

ZOPE is an object-oriented database for transparently and persistently storing Python objects.

Features of the ZODB include: transactions, history/undo, transparently pluggable storage,

built-in caching, multiversion concurrency control (MVCC) and scalability across a network.

Pyramid (http://www.pylonsproject.org/projects/pyramid/about).

Pyramid is a very general open source Python web framework. Pyramid is a minimalistic,

platform-independent object publishing web framework. It also has integration with the Zope

Object Database (ZODB).

Main objects on the ZODB used for meteorological forecasts and observation gathering are

shown below.

Figure 10 – Main objects on the Zope Object database.

GEO

Projections

Bounds

Resolution

Types

Periodicity

Actual Data

- Raster - Features

Day 1

Day 2

Day n

STORES SOURCES CONTEXTS


http://nginx.org/

http://www.zodb.org/en/latest/

http://www.pylonsproject.org/projects/pyramid/about



CIP-ICT-PSP-2012-6 – 325161




5.2.3 Data ingestion in Sigym3

Below is a detailed diagram of the FTP ingestion services and the meteorological server

modules.

Figure 11 – Data ingestion in Sigym3.

FTPRelayer, developed by Meteogrid (https://github.com/meteogrid/FTPRelayer), is a

component based on Linux’s inotify that redistributes files which arrive at a machine to

several machines, optionally processing them before sending them. It allows passive waiting

for incoming files, avoiding polling. It is freely available in the provided URL.

5.2.4 Data Access

Web Map Service (WMS) is a standard protocol for serving georeferenced map images over

the Internet that are generated by a map server using data from a GIS database.

Web Feature Service (WFS) provides an interface allowing requests for geographical features

across the web using platform-independent calls.

Proftpd

FtPRelayer

Mailprocessor

Collector

Genserver

Webserver

WMS

WFS

PostGIS

ZODB


https://github.com/meteogrid/FTPRelayer



CIP-ICT-PSP-2012-6 – 325161




Annex I - Sensor DB schema (52° North SOS 4.0)




CIP-ICT-PSP-2012-6 – 325161




Annex II – Mapping weather WFS to Sensor DB

ID WFS Name WFS Procedure SOS notes

86 CLES (VAL DI NON) VDN-F00

87 BASSANO DEL GRAPPA BAS-F00

88 PAOLA PAO-F00

89 SCHWECHAT

no more in the project

90 FERRARA FER-F00

91 ROVERETO91 ROV-F00

92 ROVERETO92 ROV-F01

93 TRENTINO93 TRN-F00






































CIP-ICT-PSP-2012-6 – 325161






























154 LAMIA154 LAM-F00





159 ZAGREB159 HRV-F00




163 RIJEKA HRV-F04 no buildings



166 SPLIT HRV-F07 no buildings

167 VARAZDIN HRV-F08

168 KRIZ HRV-F09


Date post:	12-Apr-2017
Category:	Technology
Upload:	sunshineproject
View:	452 times
Download:	0 times