Australian CIO Summit 2012: Big Data, New Physics, and Geospatial Super-Food by Tristan Sternson,...

transcript

Big Data, New Physics, and

Geospatial Super-Food

Tristan Sternson, InfoReady Managing Director

My Background – Tristan Sternson

� Past 12 years focussed purely on IM / BI Solutions

� Started InfoReady in 2008

� Prior Roles – Accenture Data Management & Architecture / IM Lead, PWC Consulting / IBM

� Personally designed and deployed and led many large IM and

� Personally designed and deployed and led many large IM and DW application in Australia and UK

� Thought leader in Information Management in Australia and APAC

� Early adopter Data Governance, Big Data, Industry Data Models, Appliance DW solutions

Who is InfoReady?

� Pure-Play Information Management and Business Intelligence Consulting firm

� Team InfoReady career IM and BI Experts

• One of the fastest growing consulting firms in Australia.

Australia. • IM Focused Tier One Consulting capability.

• Focus - people, process and technology

• Assisting companies turn valuable information into actionable intelligence.

• Strategy, Architecture, Solution Design & Delivery

Big Data Definition

Datasets that grow so large that they become difficult to work with, including; capture, storage, search, sharing, analytics, and visualization.

Benefits of working with larger and larger datasets allowing analysts to "spot business trends, prevent diseases, combat crime.”

We haven’t seen anything yet, as more devices come online, eg;mobile, airborn, logs, cameras, microphones etc…

Wikipedia - 2012

The Big Data Opportunity

Big Data – Why the hype?

� By 2015, nearly 3B people will be online, pushing the data created and shared to nearly 8 zettabytes.

� 30 billion pieces of content were added to Facebook this past month by 600M plus users.

� More than 2B videos were watched on YouTube … yesterday.

� In the US mobile phone users between the ages of 18 and 24 send an incredible 110 text messages per day.

� 32B searches were performed last month … on Twitter.

� Worldwide IP traffic will quadruple by 2015.

Business leaders frequently make decisions based on information they don’t trust, or don’t have

Business leaders say they don’’’’t have access to the information they need to do their jobs

Business Value

83% of CIOs cited “Business intelligence & Analytics” as part of their visionary plans to enhance competitiveness

do their jobs

of CEOs recognise they need to better understand information more rapidly in order to make swift decisions

Big Data Trends

80%20%

What the Industry Analysts say

Gartner predicts Big Data to be one of the top-10 strategic initiatives

for 2012

What the Industry Analysts say

Key take-aways from Analyst perspectives Gartner TDWI

Data will grow exponentially� �

Fusion of structured and unstructured data� �

The connection between big data and advanced analytics will get even stronger � �

Future users will not be able to put all useful information into a single data warehouse � �

Enterprise Intelligencevs. Enterprise Amnesia

puting

er Gro

Available Observation

Context

Trend: Organizations Are Getting Dumber

EnterpriseAmnesia

puting

er Gro

Sensemaking Algorithms

ContextWHY?

Trend: Organizations Are Getting DumberCom

puting

er Gro

Sensemaking AlgorithmsC

er Gro

Algorithms at Dead End.

You Can’t Squeeze Knowledge

Out of a Pixel.

scrila34@msn.com

No Context

Context, definition

Better understanding something by taking into

Better understanding something by taking into account the things around it.

Information in Context … and Accumulating

Job Applicant

scrila34@msn.com

Top 200Customer

Job Applicant

IdentityThief

CriminalInvestigation

The Puzzle Metaphor

� Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes and colors

� What it represents is unknown – there is no picture on hand

� Is it one puzzle, 15 puzzles, or 1,500 different puzzles?

� Some pieces are duplicates, missing, incomplete, low quality, or have been misinterpreted

� Some pieces may even be professionally fabricated lies

� Until you take the pieces to the table and attempt assembly, you don’t know what you are dealing with

Puzzling

12 pieces

1000 pieces

12 pieces100%

100 pieces10% (duplicates)

66 pieces

100%(pure noise)

First Discovery – “we found Dora?”

Sorting Algorithm

Another Puzzle …

10 Mins – Completed Dora Puzzels

Data Finds Data

Obvious Duplicates in Front Of Your Eyes

Incremental Context – Incremental Discovery

10:00am START

< 1min “I can see Dora”

1min “How many puzzles are there?”

8min “Are there 1000 pieces and 3 or 4 puzzles?”

10min 2 x Dora puzzles complete

12min “I have blue sky and an animal”

18mins “The other puzzle is more colourful – maybe a red

motorbike”

23min “we’ve found Jenny Sanders – can I search google on my

iPhone for the picture?”

35min “How can we have 2 pieces the same?”

Lots of Sorted Pieces

Pieces in Context

Quickly we find meaning (90mins)

66 piecesof

1190 piecesonly 5.5%

Wow 1%

11 piecesof

1190 piecesonly 1%

Koala, Possum or Monkey?

Foundation

More Data Finds Data

Out of Tablespace…

Incremental Context – Incremental Discovery

55min “Second puzzle is definitely a motorbike – I can see a wheel and seat”

65min Motorcycle coming together very quickly

70min “It’s definitely a koala”

75min “The koala has a baby”

83min “The middle piece of the bike is missing – do I really need it, I know what it is”

88min “These are both Australian puzzles”

114min One of the kids starts isolating pieces that are causing her “noise”

130min 7 chunks emerge from 7 piles of SORTED pieces

165min Pieces beginning to come together quite quickly and picture starts to really emerge

How Context Accumulates

� With each new observation … one of three assertions are made: 1) Un-associated; 2) placed near like neighbors; or 3) connected

� Must favor the false negative

� New observations sometimes reverse earlier assertions

� Some observations produce novel discovery

� As the working space expands, computational effort increases

� Given sufficient observations, there can come a tipping point

� Thereafter, confidence improves while computational effort decreases!

e Ident

Overstated Population

Observations

e Ident

True Population

Counting Is Difficult

Mark Smith6/12/1978

Mark R Smith(614) 13-123-123DL: 00001234

6/12/19780413123123

File 1

File 2

e Ident

The Rise and Fall of a Population

Observations

e Ident

True Population

Data Triangulation

Mark Smith6/12/1978

Mark R Smith(614) 13-123-123DL: 00001234

New Record

6/12/19780413123123

File 1

File 2

Mark Randy Smith0413123123

DL: 00001234

Big Data [in context]. New Physics.

�More data: better the predictions– Lower false positives

– Lower false negatives

�More data: bad data good– Suddenly glad your data is not perfect

�More data: less compute

Big Data

Pile of ____ In Context

One Form of Context: “Expert Counting”

� Is it 5 people each with 1 account … or is it 1 person with 5 accounts?

� Is it 20 cases of H1N1 in 20 cities … or one case reported 20 times?

case reported 20 times?

� If one cannot count … one cannot estimate vector or velocity (direction and speed).

�Without vector and velocity … prediction is nearly impossible.

Expert Counting: Degrees of Difficulty

IncompatibleFeatures

Deceit

Bob Jones123455

Ken Wells550119

Exactly Same

Bob Jones123455

Robert T Jonnes000123455

Bob Jones123455

bjones@hotmail

Key Features Enable Expert Counting

People Cars Router

Name Make Device IDAddress Model MakeDate of Birth Year ModelPhone License Plate No. Firmware Vers.Passport VIN Asset IDNationality Owner Etc.

Passport VIN Asset IDNationality Owner Etc.Biometric Etc.Etc.

Consider Lying Identical Twins

#123Sue3/3/84UberstanExp 2011

PASSPORT#123Sue3/3/84UberstanExp 2011

PASSPORT

Fingerprint

DNAMost Trusted

Authority

“Same person –

trust me.”

Most TrustedAuthority

�The same thing cannot be in two places … at the same time.

Two different things cannot

�Two different things cannot occupy the same space … at the same time.

Space & Time Enables Absolute Disambiguation

People Cars Router

Name Make Device IDAddress Model MakeDate of Birth Year ModelPhone License Plate No. Firmware Vers.Passport VIN Asset IDNationality Owner Etc.

When When WhenWhere Where Where

Passport VIN Asset IDNationality Owner Etc.Biometric Etc.Etc.

“Life Arcs” Are Also Telling

Bill Smith13/4/67

Melbourne, Victoria

Bill Smith13/4/67

Brisbane, Queensland

Address History Address History

Address History

Melbourne, Vic 2008-2008

St Kilda, Vic 2005-2008

Hampton, Vic 1996-2005

Brighton, Vic 1984-1996

Address History

Carina, QLD 2005-2009

Brisbane, QLD 2005-2005

Bondi, NSW 1990-2005

Carina, QLD 1982-1990

Space-Time-Travel

� Cell phones are generating a staggering amount of geo-locational data – 600B transactions per day being created in the US alone

� This data is being “de-identified” and shared with third parties – in volume and in real-time

third parties – in volume and in real-time

� Your movement quickly reveals where you spend your time (e.g., evenings vs. working hours)

� Re-identification (figuring out who is who) is somewhat trivial

Powerful Predictions

� Prediction with 87% certainty where you will be next Thursday at 5:35pm

�Names of the top 10 people you co-locate with, not at home and not at work

not at home and not at work

� Intelligence service preempts the next mass protest in real-time

� Robbery of a convenience store is about to happen at 10:42pm

Consequences

�Space-time-travel data is the ultimate biometric

� It will enable enormous opportunity

� It will unravel one’s secrets

� It will challenge existing notions of privacy

�And, it’s here now and more to come

Macro Trends

of Dat

aThe Greater the Context, the Greater the Value

Data in Context

of Dat

Pile of Data

Records Managed(Big) (Ludicrous Big)

Willing

The better the predictions … the faster they will be

wanted.

“Why did we have to wait until the

end of the day for the smart answer?”

Time Is Of The Essence

Willing

the smart answer?”

Relevance (Iffy) (Totally)

200ms Real-Time

Enterprise IntelligenceOne Plausible Journey

ObservationSpace

Sense and Respond

What you know

New Observations

ObservationSpace

Data Finds Data

Sense and Respond

Decide

?Relevance

Finds the Sensor(<200ms)

Data Finds Data

Explore and Reflect

ObservationSpace Deep

Reflection

CuratedData

PatternDiscovery

Data Finds Data

Sense and Respond

Decide

DirectedAttention

Relevance Find You

PatternDiscovery

RelevanceFinds the Sensor

(<200ms)

Data Finds Data

Reflection

CuratedData

PatternDiscovery

Data Finds Data

Explore and ReflectSense and Respond

Decide

DirectedAttention

NEWINTERESTS

PatternDiscovery

(<200ms)

Data Finds Data

Reflection

CuratedData

PatternDiscovery

Data Finds Data

Explore and ReflectSense and Respond

Decide

DirectedAttention

NEWINTERESTS

PatternDiscovery

(<200ms)

Data Finds Data

Report and Manage

Closing Thoughts

The most competitive organizations

are going to make sense of what they are observing

fast enough to do something about it

while they are observing it.

Context

Wish This On The Enemy

EnterpriseAmnesia

puting

er Gro

The Way Forward: Enterprise Intelligence

Context

puting

er Gro

Questions?

Email: tristan.sternson@infoready.com.au

Twitter: http://www.twitter.com/tsternson

Blog: www.infoready.com.au

LinkedIn: http://www.linkedin.com/in/tristansternson

Australian CIO Summit 2012: Big Data, New Physics, and Geospatial Super-Food by Tristan Sternson,...

Technology