INSIGHT: Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHTDermot Kinane (21st January 2015)

1

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Overview

Dublin City Council Introduction and background leading into traffic use-case

• Local Authority with a number of responsibilities

Current Situation Facilities

• Control Room (24/7)• Radio Station• AA Road Watch/Dublin Bus/Gardai• Incident Management Centre

Systems Limitations INSIGHT in Use

Next steps with INSIGHT

2

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Objective

INSIGHT

Intelligent Synthesis and Real-time Response Using Massive Streaming of Heterogeneous

Data

3

Develop an adaptive, scalable and dependable, real-time infrastructure for improving our ability to identify and cope with emergencies

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Motivation

Currently, little infrastructure in place to take advantage of data provided by sensors or humans

Improve understanding, prediction and warning of emergencies through real-time processing of data streams including social data

Guiding more informed decisions leading to effective response

4

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Overview

Dublin City Council Introduction and background leading into traffic use-case

• Local Authority with a number of responsibilities

Current Situation Facilities

• Control Room (24/7)• Radio Station• AA Road Watch/Dublin Bus/Gardai• Incident Management Centre

Systems Limitations INSIGHT in Use

Next steps with INSIGHT

5

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Traffic Control Room

6

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Traffic Control Room

7

Operations team (24/7)

Radio station

AA Roadwatch

Dublin Bus

Gardai

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Incident Management Centre

8

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Systems in use

SCATS (Sydney Coordinated Adaptive Traffic System)

CCTV

TRIPS

DPTIM

FMS

RTPI

9

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual/Unpredicted Congestion

10

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (M50 Fire)

11

Tuesday 30th September 2014~14:30

Lots of data – needs to be integrated to tell the full story

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (SCATS)

12

Auburn - Blanchardstown

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (SCATS)

13

Naas Road – Heuston Station

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (SCATS)

14

Rathfarnham Road (inbound)

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (SCATS)

15

Samuel Beckett Bridge

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (bus)

16

12:00:00 AM

12:05:00 AM

12:10:00 AM

12:15:00 AM

12:20:00 AM

12:25:00 AM

12:30:00 AM

12:35:00 AM

12:40:00 AM

12:45:00 AM

12:50:00 AM

12:55:00 AM

1:00:00 AM

1:05:00 AM

1:10:00 AM

1:15:00 AM

29.09.2014

30.09.2014

St. John’s Road

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (bus)

17

Chapelizod

0:00:00

0:05:00

0:10:00

0:15:00

0:20:00

0:25:00

0:30:00

0:35:00

0:40:00

29.09.2014

30.09.2014

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (bus)

18

Harold’s Cross

12:00:00 AM

12:05:00 AM

12:10:00 AM

12:15:00 AM

12:20:00 AM

12:25:00 AM

12:30:00 AM

12:35:00 AM

12:40:00 AM

29.09.2014

30.09.2014

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (Tweets)

19

Live Drive Tweets

Impressions – average 45,206 – fire 165,346

Replies – average 19 – fire 132

0

20

40

60

80

100

120

140

160

180

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (Tweets)

20

M50 fire

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Unusual Congestion and its Effects (Results)

21

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Limitations

Manual operation

We find out about all serious incidents… eventually

Platform doesn’t exist to integrate heterogeneous streams

Opportunities…

22

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Challenges

Handling heterogeneity

Handling large datasets

Handling uncertainty

Real-time processing

23

Partners

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Data Integration & Availability

24

- Working with DCC, eircom & IBM, a virtual private cloud (VPC) solution was proposed. Using Amazon Web Services (AWS), an IPSEC VPN to the DCC firewall was created.

- Using AWS brought the following advantages:- Compute services based on pay-as-you-use rates- 24x7 management of servers, up to OS layer included- Regular back-up schedule- All operating system licenses built into price- High resilience as it is running over 2 Availability Zones (AZ)

- Bus data retrieval is simply from a data storage area in AWS S3 (Storage Bucket)

- SCATS data stream provided in JSON

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Solutions

25

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT Architecture

26

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action (symbols & examples)

27

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

28

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

29

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

30

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

31

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

32

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

33

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

34

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

INSIGHT in action

35

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Crowdsourcing

36

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Overview

Dublin City Council Introduction and background leading into traffic use-case

• Local Authority with a number of responsibilities

Current Situation Facilities

• Control Room (24/7)• Radio Station• AA Road Watch/Dublin Bus/Gardai• Incident Management Centre

Systems Limitations INSIGHT in Use

Next steps with INSIGHT

37

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Design of Validation Experiments

Description/Validation of Experiments/Next Steps

How DCC intend to put the INSIGHT software to test

• 1st phase - Radio Station to be test bed

• To verify that events are happening

• False positives/missed events

• 2nd phase – Control Room operator(s)

• All INSIGHT partners have been invited to work from the Dublin City Council Traffic Control Room when the INSIGHT beta is released

Plan for testing crowdsourcing app in place

Extended datasets

38

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Summary

Currently a very manual operation

INSIGHT very relevant to DCC

Intelligent synthesis

Real-time response

Streaming heterogeneous data

Confidence level

Accurate deployment of resources

Opportunity to add value to DCC operation (& other cities)

Further information

http://www.insight-ict.eu/

39

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Backup slides

40

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Events & Notifications

41

Prototype – City Case

Root

Anomaly

Traffic Jam

Low Speed High DelayHigh Degree of Saturation

Flood

Normal

SCATS ISA High Degree of Saturation

Twitter ISA Flood Traffic Tweet (Traffic or Flood Related)

Bus ISA Low Speed, High Delays Noisy Sensors

Bus Reports Extremely High Speed Bus Moving but Reports that is at Bus

Stop Bus seems to be far away from the stop

it reports to be at

Crowdsourcing Tasks (questions) are sent

to the users

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

Second Year Demo Architecture

26

Prototype – City Case

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data

❑http://195.134.71.40:8000/html/visualization/