Analysis of Data Placement Strategy based on Computing Power of Nodes onHeterogeneous Hadoop Clusters
Sanket Reddy Chintapalli Advisor - Dr. Xiao Qin
Presentation Overview
● Synopsis● Mapreduce Programming Model Overview● HDFS Overview● Motivation● Design● Software Description● Hardware Description● Results● Conclusion
Synopsis
● Data placement strategy● Heterogeneous Clusters● Computing Power● Calculating Computing Ratio● WordCount and Grep
MapReduce Model
● Hadoop 1.0 and Hadoop 2.0● Master - Slave Model● JobTracker and TaskTracker Hadoop 1.0● YARN Hadoop 2.0● Resource Manager YARN● Application Manager YARN● Node Manager YARN● MapReduce Flow
Mapreduce Model
Mapreduce Model - 1.0
Mapreduce Model - YARN - 2.0
Mapreduce Model - Flow
HDFS
● Namenode● Datanode● Replication● Federated Namenodes
HDFS Architecture
HDFS Federated Namenodes
HDFS Federated Namenodes
● Scalability● Performance● Isolation - overload
Motivation
Software Description
● Hadoop 2.3.0● Maven● Eclipse● Protocol Buffers
Hardware Description
Design
Run WordCount and Grep Applications on individual nodes
Design
Calculate Computing Power of Individual Nodes fora specific application
Design
● Evaluate Hadoop Distribution by running grep and wordcount together on all nodes
● Run the CRBalancer to balance the nodes● Finally re-run the applications to note the ramifications
of the data placement strategy.
Design - Algorithm
CRBalancer Strategy
Implementation
● CRBalancer ● CRBalancingPolicy● CRNamenodeConnector
Results - WordCount
Results - Grep
Questions ??
