Date post: | 27-Jan-2015 |
Category: |
Technology |
Upload: | clearstone-venture-partners |
View: | 114 times |
Download: | 8 times |
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Store files
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Process
data
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Ingest data
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Store records & tables
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
High level domain specific
language
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Chain together complex workloads
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Schedule them
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Columnar format + metadata
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
End user queries
What did Google do?
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Dremel
Dremel Evenflow
MySQL
Gateway
Sawzall Bigtable
Chubby
MapReduce / GFS
Evenflow
Coordinate within
system
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
HiPal
Hive Databee Databee
Scribe
Hive HBase
Zookeeper
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Hive Oozie Oozie
Data
Highway
Pig & Hive HBase
Zookeeper
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Azkaban Azkaban
Sqoop
Kafka
Pig Voldemort
Zookeeper
The pattern repeated
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Hue Hue
Hive Oozie Oozie
Sqoop
Flume
Hive / Pig HBase
Zookeeper
Cloudera’s Distribution Including Apache Hadoop
Project summary
©2011 Cloudera, Inc. All Rights Reserved. Confidential.
Reproduction or redistribution without written permission is
prohibited.
Topic Project(s)
File storage HDFS
Record storage Hbase, Hypertabe, Accumulo
Metadata storage Hive, Hcatalog
Batch data processing MapReduce
Streaming data processing S4, Storm
Graph processing Giraph, X-Rime
Query language Hive
Dataflow language Pig
Database integration Sqoop
Event data collection Flume, Scribe
Test & assembly Bigtop
Distributed lock Zookeeper
Web access Hue
Workflow Oozie, Azkaban
File format Avro, RCFile, Protocol Buffers, Sequence File
BIG DATA
PO
SS
IBL
E
anything
with
is
Ce
leb
rate
Ne
xt
Satu
rday