Warp10: collect, store and manipulate sensor data - BreizhCamp - 2016 03-24

transcript

@FinistSeb @LostInBrittany#BzhCmp #Warp10

Warp 10: Collect, store and manipulate sensor data

Horacio Gonzalez Sébastien Lambour

Horacio Gonzalez

@LostInBrittany

Cityzen Data

Spaniard lost in Brittany, developer, dreamer and all-around geek

Sébastien Lambour

@FinistSeb

Cityzen Data

Runner, 2 Kids, Geek, Handyman, Polyglot JVM Developer

Warp 10

IntroductionGeo-Time SeriesTM

Image: Spacetime distorsions

Time Series

Image: Mike Bostock

Time series storage and analysis

Image: Hamza Fessi and ABC Bourse

Not suited for your vanilla SQL RDBMS

One simple example: moving averages...

Geo-Time SeriesTM

Image: AIS Vessel Tracking

Geo-Time Series

Geo-Time Series and the IoT

Image: LinkedIn

IoT means talking thingHow fast are they talking?

Internet of very introverted Things

Long range transmissions

Internet of introverted Things

Personal Area Network

Internet of shy Things

Local Area NetworkCellular Networks

Lots of shy thing generate a huge lot of data

Image: Universal Studios

Internet of chatty Things

10 000 Hz

670 000 sensors

20 000 metrics

per second

Internet of garrulous Things

Image: Google

Millions of metrics per second

Warp 10 : A software platform for IoT

Warp 10 is a software platform that● Ingests and stores data● Manipulates and analyzes data● Is dedicated to data from sensors, meters, IoT and any real or

virtual probe

Warp 10 General Synoptic

Stockage Architecture

Language, Functions, Algorithms

Application access

VizualisationRealTime

#collectHow do you get these metrics?

Image: Games Radar

Using our own Sensision agent

With queue forwarder

Using plugins for other collecting systems

Or simply pushing data directly

Choosing an input format

XML? JSON?

139 bytes 108 bytes

Warp 10 GTS Input Format

57 bytes

But size isn't the most important reason

parsing time is way more important

XML or even JSON parsing is slow and costlyWarp 10 GTS input format isn't

timestamp (us by default)

latitude:longitude (WGS84)

elevation (millimeters)

classname*

labels (key=value)

value* (long, double, boolean or string)* mandatory fields

Warp 10 GTS Input Format

#storeFrom tiny to huge

Image: Games Radar

Warp 10 on Raspberry Pi B+

1 000 datapoints per second

Warp 10 on Raspberry Pi 2 B

Warp 10 on a modern server

Warp 10 on a cluster

3 millions of datapoints per second(our current record on input traffic)

#analyseFrom tiny to huge

Image: Amazon

Many time-series solutions

But they are only stores...

Fetching data is only the tip of the iceberg

Analysing the data

High level analysis must be done elsewhere

Algorithms are resource hungry

Your computer is not a datacenter

Manipulating GTS

To be scalable, analysis must be done in Warp 10 platform, not in user's computer

A true GTS analysis toolbox○ Hundreds of functions○ Manipulation frameworks○ Analysis workflow

Manipulating GTS

GTS manipulation

Why not a simple REST API?● One endpoint by function?● How to chain a workflow analysis?

REST API not suitable for complex manipulations

GTS manipulation

Why not a SQL dialect?● How do you do a simple moving average in SQL?● How do you geo-time fencing in SQL?

SQL is not adapted to (G)TS analysis!

GTS manipulation language

Our solution: a GTS manipulation language

WarpScript

A stack based language

WarpScript

Non-compiled Optimized functions, fast execution

Basic operations

Five frameworks

●●●●●

More than 500 functions

Time series functions

Geo-Time Series functions

Geo mapping (WKT)

Horizontal & vertical speed

Horizontal & vertical distance

Haversine...

Quantum IDE

Enough teasing...

Fuel prices data

16 297 448 metrics

11 379 fuel stations

42 885 Geo Time Series

Basic analysis

Average diesel fuel prices in France since 2007

Image: LEGO Ideas

First Fetch Data (SQL vs WarpScript )

FETCH gives us a GTS list

Timestamp (microseconds since epoch)

Location (latitude, longitude)

Calculate the average

Using Groovy:

Calculate the average with WarpScript

1- Calculate the mean price by station

BUCKETIZE framework

Put the data of a GTS into regularly spaced buckets

2- Reduce to get the global average

REDUCE framework

Apply a function on a set of GTS tick by tick

Too verbose? Write it differently

Even more concise

Basic analysisMean of the last available

diesel fuel prices in France

Image: LEGO Ideas

Fetching Data (SQL vs WarpScript )

Mean of those last prices

align ticks with BUCKETIZE framework

compute the average with REDUCE

Geo-time analysis

Find the cheapest fuel station near here

48.115434, -1.636877

WKT: Well-known text geometry

WKT in WarpScript

Geo-filtering points of GTS

MAPPER framework

Apply a function on values of a GTS that fall into a sliding window

The stations near my position

There can only be one

And this is only the surfacePossibilities are endless

Think differently

Geo-Time Series are everywhere

Warp 10 platform and tools

Everything is on GitHub

https://github.com/cityzendata/

Thank you !

Warp10: collect, store and manipulate sensor data - BreizhCamp - 2016 03-24

Technology