SC7 Workshop 1: Big Data in Secure Societies

transcript

BIG DATA EUROPE

Integrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges

negative conotation

Big Data has often a

6-oct.-15www.big-data-europe.eu

Big Data in Marketing

Big Data in Intelligence

BigDataEurope aims to help maximizing

the societal value of Big Data

Health, demographic change and wellbeing;

Food security, sustainable agriculture and forestry, marine and maritime and inland water research, and the Bioeconomy;

Secure, clean and efficient energy;

Smart, green and integrated transport;

Climate action, environment, resource efficiency and raw materials;

Europe in a changing world - inclusive, innovative and reflective societies;

Secure societies - protecting freedom and security of Europe and its citizens.

The three Big Data „V“ – Variety

is often neglected

Quelle: Gesellschaft für Informatik

Proactive Maintenance at Rolls Royce

New Business Model integrating Sensor Data & Big Data Analytics

Dr. Dirk Hecker

Condition Monitoring, Proactive maintenance, „Power-by-the-hour“,

as-a-service Business Model – payment modell by flight hours

Quelle: www.springboeck.ch/SR_Technics.htm

The rolling Smartphone

New Business Models for the Automotive Industrywith Data Value Chains

Dr. Dirk Hecker

Windshield wiper as rain sensors for micro wether prognosis

Automotive industry can become data provider for other industries

tor.co

Predictive Analytics

Dr. Dirk Hecker

From Business Intelligence to Big Data Analytics

Business Intelligence Monitoring Predictive Analytics

What happenedbefore?

What happensnow?

What will happen soon?

What shouldhappen?

Prescriptive Analytics

„the last Mile“

“prescriptive analytics suggests decision options on how to take advantage of a future opportunity”

Quelle: BMW Quelle: www.7-forum.com Quelle: BMW Quelle: Volvo

BigDataEurope Rationale

Show societal value of Big Data

Lower barrrier for using big data technologies

o Required effort and resources

o Limited data science skills

o Lack of Generic Architectures, components

Help establishing cross-

lingual/organizational/domain Data Value Chains

o Multiple Data Sources

o Required: Integration, Harmonisation

BigDataEurope: Objectives

COORDINATION

Stakeholder Engagement

(Requirements Elicitation)

SUPPORT

Design, Realise, Evaluate

Big Data Aggregator

Platform

Create and Manage

Societal Big Data

Interest Groups

Cloud-deployment

Big Data Aggregator

Platform

Measures

Results

Orthogonal Dimensions of Big Data Ecosystems

Generic Big Data Enabling Technologies

Data Value Chain

Data Generation & Acquisition

Data Analysis & Processing

Data Storage & Curation

Data Visualization &

Data-driven Services

Healthcare

Food Security

Energy

Intelligent Transport

Climate & Environment

Inclusive & Reflective Societies

Secure Societies

Stakeholder Engagement Cycle

Open-Source Technologies for Big Data Apps (small selection :-)

Big Data - Technologies

Volume

VelocityVariety

Groups of Technologies

Big Data Technolog

Data Storage Technologies

Data Processin

Workflow Coordinati

Querying/

Processing

Search

Data Export/ Import

Data Analysis

Statistics

Text Mining

Big Data Requirements

Analysis of historical dta

Millions of entries

Varying analysis quesitions

Years of input data

=> Big Data Batch Processing

Interactive analysis by online queries

Thousands of users online

Extremely fast response time

Super high availability

=> Big Data Databases

Analysis of actual data with low latency in

"real-time"

React to newest trends

Low-Latency change detection

Real-time online monitoring

=> Big Data Stream Processing

But how to put it together ?17

The „traditional“ Hadoop Ecosystem + NoSQL

components

a Big Data Management System

ZooKeeper

askaban

cassandra

voldemort

MongoDB

CouchDB

elastic search

lucene

Function

Data Storage

pages with

postings

Batch View

Realtime

message passing

Application

Horizontal Scalability in the Lambda Architecture

> volume

> users

> users, volume

> velocity> volume, velocity

Blueprint of the Data Aggregator Platform

Follows typical Lambda Architecture

Integrated on top of existing Big Data distribution

+ Semantic Layer (Retaining Semantics using LD approach )

Batch Layer

Speed Layer

Data Storage

Real-time data &

Transactions …

Batch View

Real-time

message passing

Applications & Showcases

Real-time dashboards

Domain-specific BDE apps

Big Data Analytics

In-stream Mining

Intellig

Input data

Stream

Spatial

Social

Statistical

Temporal

Transactional

Imagery

BDE Platform based on BigTop

Packaging Smoke testing Virtualization

Package RPMs and DEBs, so

that you can manage and

maintain your own cluster.

Integrated smoke testing

framework

Vagrant recipes, raw images,

and docker recipes for

deploying BigData

infrastructures from zero.

6-oct.-15www.big-data-europe.eu+ Semantic Layer - Retaining Semantics using Linked Data

Data Aggregator Platform Challenges

Ingest semantic (RDF) and non-semantic (CSV,

JSON, XML, …) data

o Integrate various mapping techniques (R2RML, CSV on

the Web, JSON-LD)

preserve semantics, provenance and metadata in

Big Data processing chains

o Preserve URI/IRIs

o Preserve triples

Exploit semantics for aggregations

Current Activities – Year#1

2015 BDE Societal Workshops (7) Planned

o Schedule on Website

7 W3C Interest Groups set up: Please Join!o SC1: HEALTH https://www.w3.org/community/bde-health/join

o SC2: FOOD & AGRICULTURE https://www.w3.org/community/bde-food/

o SC3: ENERGY https://www.w3.org/community/bde-energy/

o SC4: TRANSPORT https://www.w3.org/community/bde-transport/

o SC5: CLIMATE & ENVIRONMENT https://www.w3.org/community/bde-climate/

o SC6: SOCIETIES https://www.w3.org/community/bde-societies/

o SC7: SECURITY https://www.w3.org/community/bde-secure-societies/

www.big-data-europe.eu

BDE Partners

Sören Auer

Big Data Europe Coordinator

Fraunhofer IAIS & University of Bonn

auer@cs.uni-bonn.de

Thanks

Energy/Climate Example: Greenshifting

SC7 Workshop 1: Big Data in Secure Societies

Technology