Mad skills new analysis practices for big data

transcript

MAD SKILLS: NEW ANALYSIS PRACTICES FOR BIG DATA

Big Data Has Arrived

Source : 2011 IDC Digital Universe Study

GROW 44X IN THE NEXT 10

THE DIGITAL UNIVERSE WILL

Big Data: Hype or Reality? •  Do we have a Big Data problem in New Zealand?

•  Do we have a Big Data problem in my organisation?

•  Do I really need to care?

Big Data: Hype or Reality? •  Do we have a Big Data problem in New Zealand? Maybe.

•  Do we have a Big Data problem in my organisation? Maybe.

•  Do I really need to care? ABSOLUTELY.

•  Big Data Practices is about widely applicable

Shadow systems ‘Shallow’

Business Intelligence

Static schemas accrete over

Slow- moving models

Slow- moving

Departmental warehouses

Data Sources

Images

A Common Analytics Environment

Data Warehouses, File Systems

SAS/ACCESS

Data Warehouse, File Systems

Cluster Computer

SAS/CONNECT SAS/ACCESS

Data Warehouse, File Systems

Cluster Computer

SAS/CONNECT SAS/ACCESS

Bag of Tricks - DATA Step Bravado - SQL Pushdown - SASFILE - SAS Views - Compression

A Leaner Configuration

Parallel Database

SAS/ACCESS

Greenplum ... ...

... ...

An Integrated Architecture

Parallel Database

SAS/ACCESS SAS/CONNECT

Greenplum ... ...

... ...

SAS High Performance Appliance on Greenplum

Parallel Greenplum Database

SAS/ACCESS SAS/CONNECT

Greenplum ... ...

... ...

SAS-GP High-Performance Analytics

Worker Node 1

Worker Node 2

Worker Node N

Master

Analytical Computation and data request sent to the worker nodes

Worker Node 1

Worker Node 2

Worker Node N

Master

Data request sent to the database, data slice moved into memory

Worker Node 1

Worker Node 2

Worker Node N

Master

Analytic Processing with internode communication

Worker Node 1

Worker Node 2

Worker Node N

Master

Worker node results returned to the Master Node, finalize computation

Worker Node 1

Worker Node 2

Worker Node N

Root Node

Result returned to the client

How do we sort out this mess?

Shadow systems ‘Shallow’

Business Intelligence

Static schemas accrete over

Slow- moving models

Slow- moving

Departmental warehouses

Data Sources

Images

Keep the Enterprise Data Warehouse

Enterprise Data Warehouse

•  Single Source of Truth •  Heavy data governance and quality •  Operational reporting •  Financial consolidation

Add an Analytics Data Cloud as a Complement

Commodity Hardware

Virtual Machines

Public Cloud

SAS/Greenplum/Hadoop

•  Single Source of Truth •  Heavy data governance and quality •  Operational reporting •  Financial consolidation

Analytics Data Cloud • Source of all raw data (often 10X size of EDW) • Self-service infrastructure to support multiple

marts and sandboxes • Ad hoc, business-led analytics solutions

MAD Analytics

Magnetic

ADC PLATFORM

Analytics

Chorus

Simple linear models Trend analysis

Model selection

Model design

analyze and model in the

cloud push results back into the

get data into the cloud

Facts Interpretation

What will happen?

How can we do better?

What happened where and

How and why did it happen?

Different Phases of Analytics PREDICTIVE MODEL

Linear Regression

Logistic Regression

Naïve Bayes Classifier

Decision Trees

Neural Networks

TRANSFORMATION Aggregation

Row Filtering

Deriving New Variables

Pivoting

Normalizing

DATA EXPLORATION Frequency

Histogram

Bar Chart

Box Plot Chart

Correlation Matrix Data Exploration

Data Prep Modeling Model Fit

DATA MINING

Association Rule

K-means Clustering

MODEL FIT STATISTICS Goodness of Fit

Significance statistics for all independent variables

SCORING

Linear Regression

Logistic Regression

Naïve Bayes Classifier

Decision Trees

Neural Networks

Scoring

In-Database Machine Learning •  Goal: Build models using all available data

•  Principle: Avoid using samples if possible.

•  Principle: Bring computation to data, not the other way round.

•  In practice: Write machine learning algorithms in (parallelised) data languages like SQL, SAS, and MapReduce.

Design Pattern – Online Learning •  Process data one at a time using an incrementally maintained

model; adjust model every time we make a prediction error

•  Examples: perceptron, online SVMs, Bayesian filters, etc.

•  Such algorithms can be implemented using SAS DATA steps or SQL aggregate functions

Design Pattern – Parallel Ensemble Learning •  Break a (large) dataset into i.i.d subsets residing on each node,

learn a model on each subset in parallel, and then combine the models appropriately

•  Examples: random forests, ensembles of SVMs, etc.

Design Pattern – MapReduce •  Repeatedly apply a Map function to transform (local) chunks of

data and then use a Reduce function to consolidate the transformed results

•  Examples: parallel LDA, k-Means, Naive Bayes, etc.

Design Pattern – Prediction Markets

Japanese Telco: What People Are Talking About

Traffic Network Modelling

Massively Parallel Model Learning •  Solving tens of thousands of statistical modelling problems, one for each

road in the city, in parallel: libname adc greenplm server=gplum db=traffic port=5432 user=keesiong …

proc sql; select origin, dest,

linregr(travel_time, array[peak_period(entry_time), …, origin_vol, dest_vol]) from adc.route_travel_info

group by origin,dest;

•  A model: t(x) = 466 + 7.72 peakPeriod(x) + 22.5 workDay(x) + 0.378 originVol(x) + 0.691 destVol(x)

It’s MAD, but is it Mad?

Commodity Hardware

Virtual Machines

Public Cloud

SAS/Greenplum/Hadoop

•  Single Source of Truth •  1 Logical Model •  Heavy data governance and quality •  Operational reporting •  Financial consolidation

Analytics Data Cloud • Source of all the raw data (often 10X size of

the EDW) • Self-service infrastructure to support multiple

marts and sandboxes • Rapid analytic iteration, and business led

solutions

Public Cloud Computing Services

Democratisation of Data

Helping Organizations Evolve From This…

Line Of Business User

Business

I.T. Department

Database Administrator

Business Intelligence Analyst

To This… Line Of Business User

Data Platform Administrator

Business Intelligence Analysts

Data Scientists

You Should Take On Your Journey To Big Data Analytics

Top 3 Steps

Put all your data to work.

Have a data strategy.

Model less, iterate more.

First invest in people, then technology.

“A journey of a thousand miles begins with a single step”

- Lao-tzu, Chinese philosopher (531 BC)

First Step: Walk towards the SAS-Greenplum Technical Session.

Mad skills new analysis practices for big data

Technology