BEATING THE TRAFFIC JAM USING NOSQL

NoSQL Matters

Kristoffer Dyrkorn, BEKK

22/11/14

Norwegian consulting firm

Private and public sector enterprises

Strategy, technology, digital services

370 employees

Responsible for state and county roads

Planning, construction, operation

7500 employees

Spending: € 7 Billion (2013)

BEKK Public Roads Administration

BACKGROUND

Population (mill)

Area (km2)

Roads (1000 km)

Germany 81 357 644 Spain 46 500 683 Norway 5 385 95

CONTEXT

E18 Vestfold – Undrumsdal. Photo: Hans A. Rosbach

Atlanterhavsveien, www.nasjonaleturistveger.no. Photo: Harald Mowinckel

Trollstigen, www.nasjonaleturistveger.no. Photo: Steinar Skaar

Aurlandsfjellet, www.nasjonaleturistveger.no. Photo: Bjørn Andresen

BUILDING AND MAINTAINING ROADS IS EXPENSIVE

PROPER PLANNING DEPENDS ON TRAFFIC ANALYSIS

VEHICLE COUNTS & WEIGHTS DECIDE

PRECISE REPORTS ARE NEEDED

ROADS ARE INFRASTRUCTURE

Butunellen. Photo: Knut Opeide, Statens Vegvesen

Photo: Statens Vegvesen

Photo: Statens Vegvesen

TIME IS ESSENTIAL

ROAD TRAFFIC IS DYNAMIC

UPDATED TRAFFIC INFORMATION HELPS:

•  SAFETY

•  FLOW CONTROL

•  ROUTE PLANNING

REAL TIME DATA IS NEEDED

Measure &

adjust

Intelligent Transport Systems:

road traffic

SYSTEM GOALS & REQUIREMENTS

EASE OF INSTALLATION AND VERIFICATION OF ROADSIDE EQUIPMENT

INCREASED DATA QUALITY

INCREASED DATA AVAILABILITY

ALL EVENTS MUST BE KEPT (NO PRE-AGGREGATION)

MINIMAL LATENCY

AD-HOC REPORTING

SCALABILITY

ROBUSTNESS

Sensor Data logger Application Storage

Voltage signature

OPC-UA event

Bulked data

A TRAFFIC EVENT

Sensors Sensors System Sensors

Other backends

GUI

Reports Vehicle info

System status

DATA FLOW

Java VM

OS

Application logic

Support libraries

Traffic events

Reports (CSV, SOAP)

Java VM

OS

Elasticsearch

Request/response

Data logger

N data loggers M application servers K storage servers

HTML5 GUI (HTTP, JSON)

SOLUTION ARCHITECTURE

"measure_point_number":  1601436,  "county_id":  16,  "region_id":  2,  "server_local_timestamp":  "2014-­‐10-­‐01T01:58:44.330+02:00",  "server_utc_timestamp":  "2014-­‐09-­‐30T23:58:44.330Z",  "client_utc_timestamp":  "2014-­‐09-­‐30T23:58:45.229Z",  "event_number":  2319762,  "vehicle_type":  3,  "vehicle_type_raw":  "9",  "vehicle_type_quality":  22228,  "vehicle_number":  2319762,  "speed":  80.9,  "length":  16.46,  "lane":  1,  "gap":  10  

HOW WE USE ELASTICSEARCH

BULK INDEXING, JAVA API

DATA IS INDEXED, STORED, NOT ANALYZED

TEMPORAL SHARDING

SPATIAL SHARDING

DATA CENTER-AWARE REPLICATION

NO SPECIAL OPTIMIZATIONS!

RAM, CPU, DISK

REPORTING

FOR A GIVEN TIME INTERVAL, PROVIDE:

•  TOTAL VEHICLE COUNT AND AVERAGE SPEED,

•  THE 85 AND 95 PERCENTILE SPEEDS,

•  IN EACH OF 5 LENGTH CATEGORIES: THE VEHICLE COUNT AND AVERAGE SPEED,

•  IN EACH OF 12 SPEED CATEGORIES: THE VEHICLE COUNT,

...AND ALL OF THIS FOR

•  EACH TRAFFIC LANE AT A MEASURE POINT,

•  EACH MEASURE POINT IN A REGION

VEHICLE FLOW (10 MIN), NOV 4TH 2014

VEHICLE SPEED (1 MIN), NOV 4TH 2014

SYSTEM VALUE

REPORTING:

•  MORE COST-EFFICIENT ROAD MAINTENANCE

REAL TIME:

•  ROUTING OF EMERGENCY VEHICLES

•  GENERAL TRAFFIC INFORMATION TO THE PUBLIC

•  ROUTE PLANNING ON HOLIDAYS

•  ROUTE PLANNING FOR PARCEL SERVICES

EXPERIENCES USING ELASTICSEARCH

ENSURE FITNESS-TO-PURPOSE

UPGRADE CONTINUOUSLY

REVISE THE RUN-TIME ENVIRONMENT CONTINUOUSLY

THE AGGREGATIONS MODULE IS FANTASTIC

USE TOOLING (WE LIKE KOPF)

MONITOR THE RESOURCE UTILIZATION

THE JAVA API IS SOMETIMES COMPLEX

WE ARE HAPPY!

?

THANK YOU!

Kristoffer Dyrkorn BEKK

kristoffer.dyrkorn@bekk.no