Welcome to the Age of Data

transcript

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

Welcome to the age of data!BIGDATA.BE

who am i

» Steven Noels» Founder & VP Product» Makers of Lily: Interactive Big Data

platform» Open Source / Apache Software

Foundation» co-founder bigdata.be

Houston,we have

a problem.

We’re drowning.

Drowningin aSeaofData.

Mountains of Metadata.

The firehoseof UGC.

Still, we can’t make

much sense of it.

... and we throw a lot of

it away.

We regard DATA as cost.

But data is an opportunity.

Think about it.

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 13

advertisements

recommendations

fraud detection

eyeballs

The future is fordatanerds.

This is what Big Data is about: new insights, new business.

3 issues forBIG DATA

need:more

capacity

solution:distributed

systems 1

distributed systems are hard.

2IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25

database

database data warehouse

database data warehouse analytics

data shuffling, data duplication

“Top-performing organizations are twice as likely to apply analytics to activities.”

(MIT Sloan Management Review, Winter 2011)

what is hadoop ?

DISK HDFS

MAP/REDUCE

many servers

map/reduce

» Batch-oriented» Data locality (code is shipped around)» Heavy parallellization» Process management» Append-only files

Hadoop ecosystem

» Hadoop Common» Subprojects» Flume/SQOOP: Data collection systems

for large distributed systems.» HBase: A scalable, distributed database

that supports structured data storage for large/wide tables.

» HDFS: A distributed file system that provides high throughput access to application data.

» Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.

» MapReduce: A software framework for distributed processing of large data sets on compute clusters.

» Pig: A high-level data-flow language and execution framework for parallel computation.

» ZooKeeper: A high-performance coordination service for distributed applications.

» Mahout: machine learning libraries

(PIG, HIVE)

Coordination(ZOOKEEPER)

DataIntegration

(FLUME, SQOOP)

FastRead/Write

Access(HBASE)

Languages / Compilers(PIG, HIVE)

Workflow(OOZIE)

Scheduling(oozie)

Metadata(HIVE)

UI Framework(HUE)

SDK(HUE SDK)

High-level data model / easy API indexes

Search

Dev2Dev tutoring,

integrateddeployment

andenterprise

supportusage metrics, analytics & recommen-

dations

speed layer

serving layer

batch layer

real-time big data architecture

1. store master dataset (append-only)2. compute arbitrary views

1. random access to batch views2. updated by batch layer

1. compensate for high latency of updates to serving layer2. fast, incremental algorithms3. batch layer eventually overrides speed layerstorm

Hadoop, interactive.

Analytics Interacticsbatchstatic files

interactivedata management

(RDBMS)

109-12

data managementindexing

search

profileharvesting

smartinsights

interactiveaudiencemetrics

news & media

telecom

My baby: Lily.

The start of Lily.

Thank you !for your attentionfor your questions

» steven.noels@outerthought.com

» @stevenn

Welcome to the Age of Data

Technology