JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of Apache Flink

transcript

On-Demand Data Streaming from Sensor Nodes

(ACM SoCC 2017)

A quick overview of Apache Flink

Presentation at Sep. 30, 2017

University of California, Santa Barbara

About me

• Researcher and PhD candidate at – Technische Universität Berlin (DIMA)

– German Research Center for Artificial Intelligence (DFKI) / (IAM)

• Working with Volker Markl

• Before – Master’s degree in Computer Science (KTH Stockholm and TU Belin)

– Bachelor’s degree in Applied Computer Science (DHBW Stuttgart)

– Four years at IBM in Germany and the USA

Jonas Traub jon@s-traub.com Jonas.traub@tu-berlin.de

Traub et al., Optimized On-Demand Data Streaming from Sensor Nodes, SoCC ‘17

Optimized On-Demand Data

Streaming from Sensor Nodes

Jonas Traub, Sebastian Breß, Asterios Katsifodimos, Tilmann Rabl, Volker Markl

Extended Talk for . .

Santa Clara, California,

September 25-27, 2017

The Sensor Cloud

Real-time

insights

The Sensor Cloud

Real-time

insights

Billions of sensor nodes form a sensor cloud

and provide data streams to analysis systems.

The Sensor Cloud

Real-time

insights

The Sensor Cloud

Real-time

insights

The Sensor Cloud

Real-time

insights

The Sensor Cloud – Problems

Real-time

insights

Real-time

insights

Streaming all data from billions

of sensors to all applications

with maximal frequencies is impossible

Real-time

insights

Streaming all data from billions

of sensors to all applications

with maximal frequencies is impossible

Increasing data rates

require expensive

system scale-out.

The Sensor Cloud – Solutions

Tailor Data Streams to the Demand of Applications

• Provide an abstraction to define the data demand of applications.

User-Defined Sampling Functions (UDSFs)

• Optimize communication costs while maintaining the result accuracy.

Read-Time Optimization

• Optimize communication costs while maintaining the result accuracy.

• Share sensor reads and data transfer among users and queries.

Read-Time Optimization

Multi-Query / Multi-User Optimization

A Motivating Example

Different Data Data Demands:

• Query 1 adaptively increases sampling rates when accelerating or braking.

• Query 2 requires a sample at least every 20 meters

• Query 3 requires a sample at least every 0.3s.

A Motivating Example - Evaluation

1/3 because 3 values per tuple

Architecture Overview

Sensor Read Scheduling

User-Defined Sampling Functions

Input:

Sensor read time and value Output:

Next Sensor Read Request

Input:

Sensor read time and value

Enable adaptive sampling techniques to reduce data transmission

e.g., Adam [Trihinas ‘15], FAST [Fan ‘14], L-SIP [Gaura ’13]

User-Defined Sampling Functions - Examples

Sensor Read Fusion

1) Minimize Sensor Reads and Data Transfer:

Latest possible read time

Read Time Optimization

2) Optimize Sensor Read Times:

● Minimize penalty while executing the minimum number of sensor reads only

● Challenge: assign read requests to sensor reads

Assigning Read Requests to Sensor Reads

Postpone Assign to next Read

Local Filtering

● Enable adaptive filtering in combination with adaptive sampling

● Enable model-driven data acquisition

Local Filtering

Evaluation

● Replay sensor data

- from a football match [DEBS Grand Challenge ’13]

- formula 1 telementry data

● Random UDSFs:

- Read in a poisson process (also simulate load peaks)

- In average 1 read per query per second

- Exponentially distributed read time tolerance

- high probability for small tolerances

- small probability for large tolerances

- In average 0.04s read time tolerance

Traub et al., Optimized On-Demand Data Streaming from Sensor Nodes, SoCC ‘17 52

Increasing the number of concurrent queries

• On-Demand scheduling reduces sensor reads and data transfer by up to 87%.

• The # of reads and transfers increases sub-linearly with the # of queries.

Increasing the number of concurrent queries

• Our read-time optimizer reduces the deviation from desired read times

by up to 69% (preserving the min. # of reads and transfers).

Increasing read time tolerances

Query Prioritization (1/2)

Query Prioritization (2/2)

Slack Robustness of Adaptive Sampling

Optimized On-Demand Data

Streaming from Sensor Nodes

Wrap-Up:

• Define data demand: User-Defined Sampling Functions

• Schedule sensor reads and data transfer on-demand

• Optimize read times globally - for all users and queries

Jonas Traub, Sebastian Breß, Asterios Katsifodimos, Tilmann Rabl, Volker Markl

A quick overview of Apache Flink - research summary -

Jonas Traub visiting September 30, 2017

Outline

Apache Flink Primer

• Stratosphere – The origin of Apache Flink

• What is Apache Flink? – Basic System Internals

• The Flink Community

An Apache Flink Research Summary

62 © Volker Markl

• Relational Algebra

• Declarativity

• Query Optimization

• Robust Out-of-core

• Scalability

• User-defined

Functions

• Complex Data Types

• Schema on Read

Draws on

Database Technology

Draws on

MapReduce Technology

Stratosphere: General Purpose

Programming + Database Execution

63 © Volker Markl

• Relational Algebra

• Declarativity

• Query Optimization

• Robust Out-of-core

• Scalability

• User-defined

Functions

• Complex Data Types

• Schema on Read

• Iterations

• Advanced Dataflows

• General APIs

• Native Streaming

Draws on

Database Technology

Draws on

MapReduce Technology

Stratosphere: General Purpose

Programming + Database Execution

64 © Volker Markl 64

64 © Volker Markl

Apache Flink is an open source platform for scalable batch and stream

data processing.

What is Apache Flink?

http://flink.apache.org

A distributed system that you can use to process data

Like a DBMS but not exactly a DBMS

What kind of data? Data that comes in the form of streams

What kind of processing

Quite flexible. You can use Java/Scala APIs similar to programming with Java collections, the new SQL API, etc

Distributed: runs on many (1000s) of machines and hides this complexity from the user

65 © Volker Markl

Basic application architecture

app state

event log

service

Sources of data

(e.g., sensors,

logs, …)

A replayable log of

events with pub/sub

functionality

Processing of

events

Storage and query systems

By courtesy of Kostas Tzoumas 65

Technology inside Flink

case class Path (from: Long, to: Long) val tc = edges.iterate(10) { paths: DataSet[Path] => val next = paths .join(edges) .where("to") .equalTo("from") { (path, edge) => Path(path.from, edge.to) } .union(paths) .distinct() next } Program

case class Path (from: Long, to: Long) val tc = edges.iterate(10) { paths: DataSet[Path] => val next = paths .join(edges) .where("to") .equalTo("from") { (path, edge) => Path(path.from, edge.to) } .union(paths) .distinct() next }

Cost-based

optimizer

Type extraction

Pre-flight (Client) Program

Cost-based

optimizer

Type extraction

Pre-flight (Client) DataSourc

e orders.tbl

Filter

Map DataSourc

e lineitem.tbl

Join Hybrid Hash

HT probe

hash-part [0] hash-part [0]

GroupRed

forward

Program

Dataflow

Cost-based

optimizer

Type extraction

scheduling

Recovery

metadata

Pre-flight (Client)

Master Workers

DataSourc

e orders.tbl

Filter

Map DataSourc

e lineitem.tbl

Join Hybrid Hash

HT probe

hash-part [0] hash-part [0]

GroupRed

forward

Program

Dataflow

deploy

operators

intermediate

results

Flink community

Feb15 Dec15 Dec16

NumberofContributors

Feb15 Dec15 Dec16

StarsonGitHub

Feb15 Dec15 Dec16

ForksonGitHub

By courtesy of Kostas Tzoumas

Project Statistics (Updated: Sep 29, 2017)

Companies Using Flink

Apache Flink - Related Publications

System Paper 2015:

System Paper 2014:

System Paper 2015:

System Paper 2014:

System Paper 2015:

System Paper 2014:

State Management

VLDB 2017

Iterative Processing

VLDB 2012

Iterative Processing

VLDB 2012

SIGMOD 2013 77

Fault Tolerance

Streaming Window Aggregation

Visualization of Streaming Data

On-Demand Data Streaming from Sensor Nodes

(ACM SoCC 2017)

A quick overview of Apache Flink

Presentation at Sep. 30, 2017

University of California, Santa Barbara

JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of Apache Flink

Software