(BDT207) Use Streaming Analytics to Exploit Perishable Insights | AWS re:Invent 2014

Using Streaming Analytics To

Exploit Perishable Insights

Mike Gualtieri, Principal AnalystNovember 2014

© 2014 Forrester Research, Inc. Reproduction Prohibited 2

7%

16%

15%

18%

20%

24%

5%

12%

18%

17%

16%

33%

Social related projects

Mobile related projects

Cloud related projects

Systems of engagementapplications

Systems of record applications

Data related projects

2nd PriorityTop Priority

Source: Forrsights Software Survey, Q4 2013, Base: 2,074 IT executives and technology decision-makers

Please rank the following technologies according to their importance and investment within your firm?

Executives and technology decision-makers are remembering the power of data.

Why?

Meet demand for analytics of all kinds.

#Analytics

Businesses often think of analytics as a set of

historical reports and dashboards…

Future

History

…but, analytics is also about the future.

Momentum is strongest for streaming and predictive analytics, but underadopted.

10%

13%

16%

18%

21%

21%

24%

29%

33%

37%

49%

50%

56%

58%

81%

15%

21%

20%

20%

27%

32%

19%

33%

42%

35%

50%

54%

59%

57%

77%

Non Modeled Data Exploration And…

Streaming Analytics

Advanced Visualization

Text Analytics

Metadata Generated Analytics

Predictive Analytics

Search/Interactive Discovery

Location Analytics

Process Analytics

Olap

Embedded Analytics

Performance analytics

Web Analytics

Dashboard

Reporting

2014

2012

“What is your firm's/business unit's current use of the following technologies?”

Source: Forrester Research

+62%

+52%


What are the main business and technical requirements or inadequacies of earlier-generation business

intelligence technologies that lead you to consider new BI techniques and technologies?

Base: 452 North American technology decision-makers

Respondents answering “don’t know” are not shown

Source: Global Data and Analytics Survey, 2014

Base: 249 North American business decision-makers



2%

16%

20%

28%

29%

31%

32%

32%

34%

35%

45%

2%

12%

14%

26%

23%

35%

28%

31%

27%

33%

44%

Other (please specify)

Earlier-generation technology is too expensive

The velocity of data is too high for earlier technologies

The number of data formats that we must be able to…

Analysis requirements change too fast to keep up with

The performance of certain analysis is not sufficient

We don't know what our entire data universe contains,…

We want to access data that was not accessible for us…

Data changes or becomes available much faster than…

Data volumes have grown beyond what we can cost-…

We want deeper insights through advanced analytics

Business decision makers

Technology decision makers

Most want deeper insights through advanced analytics but familar challenges persist.


What percentage of enterprise data do firms use for analytics?

A. 12%

B. 34%

C. 53%

D. 76%

Enterprise

Data

Quiz


What percentage of enterprise data do firms use for analytics?

A. 12%

B. 34%

C. 53%

D. 76%

Enterprise

Data

Quiz


#Predictive

Trend

Data science can find hidden new

knowledge and predictive models

Predictive analytics means faster decisions

10%

13%

16%

18%

21%

21%

24%

29%

33%

37%

49%

50%

56%

58%

81%

15%

21%

20%

20%

27%

32%

19%

33%

42%

35%

50%

54%

59%

57%

77%


Streaming Analytics


Text Analytics




Location Analytics

Process Analytics

Olap

Embedded Analytics


Web Analytics

Dashboard

Reporting

2014

2012



+52%


› Predictive models are about probabilities, not absolutes

• E.g. 78% chance you will like Breaking Bad

› Predictive models may not exist for every question

• E.g. Economists, elections, etc…

Predictive models can be very powerful and profitable, but understand that:

But, when they work they give your firm an “unfair” advantage.


Data scientists use a

combination of statistical

and machine learning

algorithms to find

patterns and predictive

models.


Data science is very different from traditional analytics

Traditional Analytics Predictive Analytics

• Choose a business outcome to improve

• Discuss and decide what data will be relevant

• Develop a data model

• Design reports and dashboards

• Choose business outcome to improve

• Assemble all possible data

• Run algorithms to find relevant data & predictive model

• Use the predictive model

How can Spotify use accelerometer data

generated by customers while they listen?

Activity

#BigData

Trend

Big Data means all your enterprise

data + IoT data


30%

7%

12%

21%

30%

9% 8%

14%

35% 34%

The term “big data” is very confusing; not

sure what it means

It’s a bunch of hype with little substance and few new ideas

It’s about new technologies that allow us to handle

more data

It’s an extension of existing analytics and

BI practices suited for data that is larger or faster than we are

used to

It’s a whole new way of thinking about the

value in data that requires new analytics and

leverages some new technologies

Business Decision Makers Technology Decision Makers

Base: 452 North American technology decision-makers



Base: 249 North American business decision-makers



Most technology decision makers get it; 30% of business decision makers are confused.

110010011011001

010010011011001

010011001101101

010010011011001

Historical

Transactions

Customer data

Ops

22

Gather all your data to breakdown silos and

prepare it for deeper analysis.

Data

Now, analyze the heck out of it - every which way.

Process

#TooLate

Trend

Live data is flowing by, and value is

slipping away.

Big data isn’t just about lakes…

. . . it’s also about raging torrents of data


› Ingested and stored in a data

warehouse

› Multiple sources of data

› Analytics run weekly, daily, or

hourly

› Insights used to modify future

actions

Analyzing data lakes versus streamsStreamsLakes

› Does collect data in realtime

› Multiple sources of data

› Immediately fed to streaming

application

› Analytics run continuously, second and

subsecond responses

› Insights used to proactively adjust

immediate and future actions

#Streaming

Trend

Streaming data is flowing by, and value

is slipping away.

Streaming analytics means real-time, actionable insights

10%

13%

16%

18%

21%

21%

24%

29%

33%

37%

49%

50%

56%

58%

81%

15%

21%

20%

20%

27%

32%

19%

33%

42%

35%

50%

54%

59%

57%

77%


Streaming Analytics


Text Analytics




Location Analytics

Process Analytics

Olap

Embedded Analytics


Web Analytics

Dashboard

Reporting

2014

2012



+62%

DEFINITION

FORRESTERSoftware that can filter, aggregate, enrich, and

analyze a high throughput of data from

disparate live data sources to visualize business

in real time, detect urgent situations, and

automate immediate actions.


We call these in-the-moment advantages:

#PerishableInsights

Insights that can provide incredible value but the value

expires and evaporates once the moment is gone.


Streaming analytics is only half about ingestion

› High-throughput, uneven ingestion of event,

sensor, transactions and just about any periodic

data that just flows unrequested

• Architectural concerns such as availability,

scalability, and latency (performance) are handled

by platform

• Connect to multiple live disparate data sources

35


The distinguishing magic of streaming analytics is about streaming operators

› Simple and complex analytical operators

• Detect, filter, and/or aggregate events

• Lightweight transformations and enrichment

• Dimensional window operators (e.g. break a

geofence, average pressure over 5 minutes)

• Temporal pattern detection (e.g. if A and then B

within 2 seconds)

36

Successful streaming analytics

programs bring disparate data sources

together.


The constructs of streaming applications are different from conventional applications…

Filtering

Aggregation/correlation

Enrichment

Location/motion

Time windows

Temporal patterns

Familiar

Unfamiliar


Data

WarehouseAnalyticsHistorical

…and, so is the application architecture

Push notifications

Email alerts

HTML5

Dashboards/

visualizations

APIs

Streaming

Analytics

Application

Platform

Stream 2

Backend

Database

Traditional

App

API calls/responses

Stream 1

Stream 3


Streaming analytics platforms enable a whole new class of applications

› Detect , adapt, and act applications require high-

performance data access on the front-end and the back-

end.

› Provide development tools to create streaming

applications

› Reduce development time by simplifying the

architectural concerns of performance, scalability, and

availability.

#IoT

Applications are blind – use sensors to make

them see.


If you can measure it and it’s connected to the Internet, then you can use it


Ubiquitous computing

Everyware

Ambient intelligence

Smart world

Connected worldCognitive computing

Pervasive computing

Physical computing

Context-aware pervasive systems

Machine-to-machineIndustrial Internet

Internet of everythingThingternet

Sensor revolution

#Cloud

The cloud is perfect for streaming analytics.

Trend


Lots of streaming data is cloud born

› Mobile, web and IoT data

› Elasticity of architecture can handling the

spikeyness of both ingestion and streaming

operator compute

› Lower-latency integration with other services to

enrich streams from database, data warehouses

#Challenges

Garbage In = Garbage Out


Thinking in streams requires new technology and mindset› Data silos hinder visibility and prevent streaming insights.

› Big Data strains the ability of legacy technology to delivery

handle large, uneven flows of data

› Predictive and streaming analytics capabilities are lacking or

non-existent

› There’s a lack of external and contextual data sources that

enrich data.

› Developers still think in request/response

#StreamingApps

Streaming apps anticipate a customer’s intent

and adapts to serve them.

Trend


Streaming analytics can uniquely enable three new tiers of app functionality

Source: April 22, 2014, “Use Sensors To Take Apps To The Next Level Of Customer Engagement” Forrester report

Stop

What if you knew your customer was near your

store on a sunny day?


NFL

Sensors in every players’ shoulder pads will

change the way we analyze and watch the game

How can sensors in police-issued firearms

improve safety and response?


Trip

1

Easy. Buy a copper tube ice maker kit.

Trip

2

Buy a shut-off valve for the copper

tubing.

Trip

3

Buy a T-connector to tap the cold water

supply line.

Trip

4

Whoops. Also need to buy a hacksaw to

cut the copper pipe.

Trip

5

Finally. A special drill bit to make a hole

in the kitchen floor for the copper tubing.

Streaming apps can make your

customers feel intensely loyal.


Design principles for customer –facing streaming apps

› Learning who the customer really is

› Detect the customer’s intent in-the-moment

› Adapt functionality and content to match intent

› Optimize for the device (human-computer

interface)

#Opportunity

Trend

The velocity of business requires

streaming analytics.

#Imagination

What kinds of apps could

you develop if could

predict, detect and adapt

to what is happening in

your business in-the-

moment?

Thank youMike Gualtieri

[email protected]

Date post:	30-Jun-2015
Category:	Data & Analytics
Upload:	amazon-web-services
View:	12,356 times
Download:	3 times

(BDT207) Use Streaming Analytics to Exploit Perishable Insights | AWS re:Invent 2014

Data & Analytics