Welcome to the Age of Data

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

Welcome to the age of data!BIGDATA.BE


who am i

» Steven Noels» Founder & VP Product» Makers of Lily: Interactive Big Data

platform» Open Source / Apache Software

Foundation» co-founder bigdata.be

2


Houston,we have

a problem.


We’re drowning.


Drowningin aSeaofData.


Mountains of Metadata.


The firehoseof UGC.


Still, we can’t make

much sense of it.


... and we throw a lot of

it away.


We regard DATA as cost.


But data is an opportunity.


Think about it.

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 13

advertisements


recommendations


fraud detection


eyeballs


churn


The future is fordatanerds.


This is what Big Data is about: new insights, new business.


3 issues forBIG DATA


moore

data

need:more

capacity

time

volu

me 1


solution:distributed

systems 1


1


distributed systems are hard.

1

2IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25

database


database data warehouse


database data warehouse analytics


database data warehouse analytics

data shuffling, data duplication


“Top-performing organizations are twice as likely to apply analytics to activities.”

(MIT Sloan Management Review, Winter 2011)

3


enter



HBase


Disk

CPU

RAM1

serv

er

what is hadoop ?


RAM

CPU

DISK HDFS

MAP/REDUCE

HBASE

many servers


map/reduce


map/reduce

32

» Batch-oriented» Data locality (code is shipped around)» Heavy parallellization» Process management» Append-only files


Hadoop ecosystem

» Hadoop Common» Subprojects» Flume/SQOOP: Data collection systems

for large distributed systems.» HBase: A scalable, distributed database

that supports structured data storage for large/wide tables.

» HDFS: A distributed file system that provides high throughput access to application data.

» Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.

» MapReduce: A software framework for distributed processing of large data sets on compute clusters.

» Pig: A high-level data-flow language and execution framework for parallel computation.

» ZooKeeper: A high-performance coordination service for distributed applications.

» Mahout: machine learning libraries

http://hadoop.apache.org/chukwa/

http://hadoop.apache.org/chukwa/

http://hadoop.apache.org/hbase/

http://hadoop.apache.org/hbase/

http://hadoop.apache.org/hdfs/

http://hadoop.apache.org/hdfs/

http://hadoop.apache.org/hive/

http://hadoop.apache.org/hive/

http://hadoop.apache.org/mapreduce/

http://hadoop.apache.org/mapreduce/

http://hadoop.apache.org/pig/

http://hadoop.apache.org/pig/

http://hadoop.apache.org/zookeeper/

http://hadoop.apache.org/zookeeper/


(PIG, HIVE)

Coordination(ZOOKEEPER)

DataIntegration

(FLUME, SQOOP)

FastRead/Write

Access(HBASE)

Languages / Compilers(PIG, HIVE)

Workflow(OOZIE)

Scheduling(oozie)

Metadata(HIVE)

UI Framework(HUE)

SDK(HUE SDK)

High-level data model / easy API indexes

Search

CDH

Dev2Dev tutoring,

integrateddeployment

andenterprise

supportusage metrics, analytics & recommen-

dations


speed layer

serving layer

batch layer

real-time big data architecture

1. store master dataset (append-only)2. compute arbitrary views

1. random access to batch views2. updated by batch layer

1. compensate for high latency of updates to serving layer2. fast, incremental algorithms3. batch layer eventually overrides speed layerstorm


Hadoop, interactive.

Analytics Interacticsbatchstatic files

1018

interactivedata management

1015

(RDBMS)

109-12


data managementindexing

search

profileharvesting

smartinsights

interactiveaudiencemetrics

news & media

telecom

finan

ce

com

mer

ce

My baby: Lily.


The start of Lily.


Thank you !for your attentionfor your questions

» [email protected]

» @stevenn

Date post:	15-Jan-2015
Category:	Technology
Upload:	ngdata
View:	1,393 times
Download:	1 times

Welcome to the Age of Data

Technology