Mad skills new analysis practices for big data

1 © Copyright 2012 EMC Corporation. All rights reserved.

MAD SKILLS: NEW ANALYSIS PRACTICES FOR BIG DATA


Big Data Has Arrived

Source : 2011 IDC Digital Universe Study

GROW 44X IN THE NEXT 10

YEARS

THE DIGITAL UNIVERSE WILL


Big Data: Hype or Reality? •  Do we have a Big Data problem in New Zealand?

•  Do we have a Big Data problem in my organisation?

•  Do I really need to care?


Big Data: Hype or Reality? •  Do we have a Big Data problem in New Zealand? Maybe.

•  Do we have a Big Data problem in my organisation? Maybe.

•  Do I really need to care? ABSOLUTELY.

•  Big Data Practices is about widely applicable


Today

Shadow systems ‘Shallow’

Business Intelligence

Static schemas accrete over

time

Slow- moving models

Slow- moving

data

Departmental warehouses

Data Sources

Images

EDW


A Common Analytics Environment

Data Warehouses, File Systems

SAS/ACCESS



Data Warehouse, File Systems

Cluster Computer

SAS/CONNECT SAS/ACCESS



Data Warehouse, File Systems

Cluster Computer

SAS/CONNECT SAS/ACCESS

Bag of Tricks - DATA Step Bravado - SQL Pushdown - SASFILE - SAS Views - Compression


A Leaner Configuration

Parallel Database

SAS/ACCESS

Greenplum ... ...

... ...


An Integrated Architecture

Parallel Database

SAS/ACCESS SAS/CONNECT

Greenplum ... ...

... ...


SAS High Performance Appliance on Greenplum

Parallel Greenplum Database

SAS/ACCESS SAS/CONNECT

Greenplum ... ...

... ...


SAS-GP High-Performance Analytics

Worker Node 1

Worker Node 2

Worker Node N

Master

Analytical Computation and data request sent to the worker nodes



Worker Node 1

Worker Node 2

Worker Node N

Master

Data request sent to the database, data slice moved into memory



Worker Node 1

Worker Node 2

Worker Node N

Master

Analytic Processing with internode communication



Worker Node 1

Worker Node 2

Worker Node N

Master

Worker node results returned to the Master Node, finalize computation



Worker Node 1

Worker Node 2

Worker Node N

Root Node

Result returned to the client


How do we sort out this mess?

Shadow systems ‘Shallow’

Business Intelligence

Static schemas accrete over

time

Slow- moving models

Slow- moving

data

Departmental warehouses

Data Sources

Images


Keep the Enterprise Data Warehouse

Enterprise Data Warehouse

•  Single Source of Truth •  Heavy data governance and quality •  Operational reporting •  Financial consolidation


Add an Analytics Data Cloud as a Complement

Commodity Hardware

Virtual Machines

Public Cloud

SAS/Greenplum/Hadoop


•  Single Source of Truth •  Heavy data governance and quality •  Operational reporting •  Financial consolidation

Analytics Data Cloud • Source of all raw data (often 10X size of EDW) • Self-service infrastructure to support multiple

marts and sandboxes • Ad hoc, business-led analytics solutions


MAD Analytics


Magnetic

ADC PLATFORM

Data

Analytics

Chorus

Agile

Dat

a Fa

st E

TL/E

LT

Simple linear models Trend analysis

Model selection

Model design


Agile

analyze and model in the

cloud push results back into the

cloud

get data into the cloud


Deep

Pas

t

Fut

ure

Facts Interpretation

What will happen?

How can we do better?

What happened where and

when?

How and why did it happen?


Different Phases of Analytics PREDICTIVE MODEL

Linear Regression

Logistic Regression

Naïve Bayes Classifier

Decision Trees

Neural Networks

TRANSFORMATION Aggregation

Row Filtering

Deriving New Variables

Pivoting

Normalizing

DATA EXPLORATION Frequency

Histogram

Bar Chart

Box Plot Chart

Correlation Matrix Data Exploration

Data Prep Modeling Model Fit

DATA MINING

Association Rule

K-means Clustering

MODEL FIT STATISTICS Goodness of Fit

ROC

Significance statistics for all independent variables

SCORING

Linear Regression

Logistic Regression

Naïve Bayes Classifier

Decision Trees

Neural Networks

Scoring


In-Database Machine Learning •  Goal: Build models using all available data

•  Principle: Avoid using samples if possible.

•  Principle: Bring computation to data, not the other way round.

•  In practice: Write machine learning algorithms in (parallelised) data languages like SQL, SAS, and MapReduce.


Design Pattern – Online Learning •  Process data one at a time using an incrementally maintained

model; adjust model every time we make a prediction error

•  Examples: perceptron, online SVMs, Bayesian filters, etc.

•  Such algorithms can be implemented using SAS DATA steps or SQL aggregate functions


Design Pattern – Parallel Ensemble Learning •  Break a (large) dataset into i.i.d subsets residing on each node,

learn a model on each subset in parallel, and then combine the models appropriately

•  Examples: random forests, ensembles of SVMs, etc.


Design Pattern – MapReduce •  Repeatedly apply a Map function to transform (local) chunks of

data and then use a Reduce function to consolidate the transformed results

•  Examples: parallel LDA, k-Means, Naive Bayes, etc.


Design Pattern – Prediction Markets


Japanese Telco: What People Are Talking About

!


Traffic Network Modelling


Massively Parallel Model Learning •  Solving tens of thousands of statistical modelling problems, one for each

road in the city, in parallel: libname adc greenplm server=gplum db=traffic port=5432 user=keesiong …

proc sql; select origin, dest,

linregr(travel_time, array[peak_period(entry_time), …, origin_vol, dest_vol]) from adc.route_travel_info

group by origin,dest;

•  A model: t(x) = 466 + 7.72 peakPeriod(x) + 22.5 workDay(x) + 0.378 originVol(x) + 0.691 destVol(x)


It’s MAD, but is it Mad?

Commodity Hardware

Virtual Machines

Public Cloud

SAS/Greenplum/Hadoop


•  Single Source of Truth •  1 Logical Model •  Heavy data governance and quality •  Operational reporting •  Financial consolidation

Analytics Data Cloud • Source of all the raw data (often 10X size of

the EDW) • Self-service infrastructure to support multiple

marts and sandboxes • Rapid analytic iteration, and business led

solutions


Public Cloud Computing Services


Democratisation of Data






Helping Organizations Evolve From This…

Line Of Business User

Business

I.T. Department

Database Administrator

Business Intelligence Analyst


To This… Line Of Business User

Data Platform Administrator

Business Intelligence Analysts

Data Scientists


You Should Take On Your Journey To Big Data Analytics

Top 3 Steps


3.

Put all your data to work.


2.

Have a data strategy.

Model less, iterate more.


1.

First invest in people, then technology.


“A journey of a thousand miles begins with a single step”

- Lao-tzu, Chinese philosopher (531 BC)

First Step: Walk towards the SAS-Greenplum Technical Session.

Date post:	20-Jan-2015
Category:	Technology
Upload:	sunz-sas-users-of-new-zealand
View:	954 times
Download:	1 times

Mad skills new analysis practices for big data

Technology