Date post: | 27-Jan-2015 |
Category: |
Technology |
Upload: | sqlservercoil |
View: | 1,136 times |
Download: | 3 times |
LP Facts
8500+ customers
450M unique visitors per month
1.3B visits per month
60TB of new data every month
LP Facts
Oracle – B2B SQL Server – B2C
Hadoop – Raw Data Vertica – Forecasting / BI
Cassandra – Application HA MySQL - Segmentation
MySQL NDB - ETL MongoDB – Predictive Targeting
Databases in Liveperson
Hadoop in Liveperson
• 2TB Of data streamed into Hadoop each day
• 100+ Data Nodes serving our data needs
• DataNodes are 36GB RAM, 1TBx12 SATA disks, 2 quad CPU
• Dozens (and growing) of daily MR jobs
• 5 different project (and growing) based on Hadoop eco-system
Hadoop
What is Hadoop?
• Open Source project from Apache
• Able to store and process large amounts of data. Including not only structured
data but also complex, unstructured data as well.
• Hadoop is not actually a single product but instead
a collection of several components
• Commodity hardware. Low software and hardware costs
• Shared nothing machines - Scalability
Example Comparison: RDBMS vs. Hadoop
Typical Traditional RDBMS Hadoop
Data Size Gigabytes Petabytes
Access Interactive and Batch Batch – NOT Interactive
Updates Read / Write many times Write once, Read many times
Structure Static Schema Dynamic Schema
Scaling Nonlinear Linear
Query Response Time
Can be near immediate Has latency (due to batch processing)
Hadoop Distributed File System - HDFS
• HDFS has three types of Nodes
• Namenode (MasterNode)• Distribute files in the cluster• Responsible for the replication
between the datanodes and for file blocks location
• Datanodes• Responsible for actual file store• Serving data from files(data) to client
• BackupNode (version 0.23 and up)• It’s a backup of the NameNode
Hadoop Ecosystem
• MapReduce - Framework for writing/executing algorithms• Hbase – Distributed, versioned, column-oriented database • Hive - SQL like interface for large datasets stored in HDFS.• Pig - Pig consists on a high-level language (Pig Latin) for
expressing data analysis programs• Sqoop - (“SQL-to-Hadoop”) is a straightforward tool. Able to
Imports/export tables or databases in HDFS/Hive/Hbase
MapReduce
Example: $HADOOP_HOME/bin/hadoop jar @HADOOP_HOME/hadoop-streaming.jar \
- input myInputDirs \
- output myOutputDir \
- mapper /bin/cat \
- reducer /bin/wc
What is MapReduce?
• Runs programs (jobs) across many computers• Protects against single server failure by re-run failed steps.• MR jobs can be written in Java, C, Phyton, Ruby and etc • Users only write Map and Reduce functions
• MAP - Takes a large problem and divides into sub problems.
Performs the same function on all subsystems• REDUCE - Combine the output from all sub-problems
Hbase
What is Hbase and why should you use HBase?
• Huge volumes of randomly accessed data.• There is no restrictions on column numbers for rows it’s dynamic.• Consider HBase when you’re loading data by key, searching data by key (or
range), serving data by key, querying data by key or when storing data by row that doesn’t conform well to a schema.
Hbase dont’s?• It doesn’t talk SQL, have an optimizer, support in transactions or joins. If
you don’t use any of these in your database application then HBase could very well be the perfect fit.
Example: create ‘blogposts’, ‘post’, ‘image’ ---create table put ‘blogposts’, ‘id1′, ‘post:title’, ‘Hello World’ ---insert value put ‘blogposts’, ‘id1′, ‘post:body’, ‘This is a blog post’ ---insert value put ‘blogposts’, ‘id1′, ‘image:header’, ‘image1.jpg’ ---insert value
get ‘blogposts’, ‘id1′ ---select records
Hive
What is Hive?
• Built on the MapReduce framework so it generates M/R jobs behind• Hive is a data warehouse that enables easy data summarization and
ad-hoc queries via an SQL-like interface for large datasets stored in HDFS/Hbase.
• Have partitioning and partition swapping • Good for random sampling Example:
CREATE EXTERNAL TABLE vs_hdfs (site_id string,session_id string,time_stamp bigint,visitor_id bigint,row_unit string,evts string,biz string,plne string,dims string)partitioned by (site string,day string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001'STORED AS SEQUENCEFILE LOCATION '/home/data/';
select session_id, get_json_object(concat(tttt, "}"), '$.BY'), get_json_object(concat(tttt, "}"), '$.TEXT') from ( select session_id,concat("{", regexp_replace(event, "\\[\\{|\\}\\]", ""), "}") tttt from ( select session_id,get_json_object(plne, '$.PLine.evts[*]') pln from vs_hdfs_v1 where site='6964264' and day='20120201' and plne!='{}' limit 10 ) t LATERAL VIEW explode(split(pln, "\\},\\{")) adTable AS event )t2
Pig
What is Pig?
• Data flow processing • Uses Pig Latin query language • Highly parallel in order to distribute data processing across many
servers• Combining multiple data sources (Files, Hbase, Hive)
Example:
Sqoop
What is Sqoop?
• It’s a command line tool for moving data between HDFS and relational database systems.
• You can download drivers for Sqoop from Microsoft and• Import Data/Query results from SQL Server to Hadoop.• Export Data from Hadoop to SQL Server.
• It’s like BCP
Example:
$bin/sqoop import --connect 'jdbc:sqlserver://10.80.181.127;username=dbuser;password=dbpasswd;database=tpch' \
--table lineitem --hive-import
$bin/sqoop export --connect 'jdbc:sqlserver://10.80.181.127;username=dbuser;password=dbpasswd;database=tpch' --table lineitem --export-dir /data/lineitemData
Other projects in Hadoop
There is many other projects we didn’t talk about
• Chukwa • Mahout• Avro • Zookeeper • Fuse • Flume• Oozie• Hue• Hiho• ………………..
Hadoop and Microsoft
• Bring Hive data directly to Excel through the Microsoft Hive Add-in for Excel• Build a PowerPivot/PowerView on top of Hive• Instead of manually refreshing a PowerPivot workbook based on Hive on
their desktop, users can use PowerPivot for SharePoint to schedule a data refresh feature to refresh a central copy shared with others, without worrying about the time or resources it takes.
• BI Professionals can build BI Semantic Model or Reporting Services Reports on Hive in SQL Server Data tools
HOW DO I GET STARTED
• Microsoft • https://www.hadooponazure.com/• http://www.microsoft.com/download/en/details.aspx?id=27584 (sqoop driver)
• Open Source: http://hadoop.apacha.org
• Vendors• http://www.cloudera.com • http://hortonworks.com• http://mapr.com
@
Demo
Questions?