Date post: | 11-May-2015 |
Category: |
Documents |
Upload: | mohammad-mustaqeem |
View: | 9,509 times |
Download: | 0 times |
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
MapReduce in Cloud Computing
Mohammad MustaqeemM.Tech 2nd Year
Reg No: 2011CS17
Computer Science and Engineering DepartmentMotilal Nehru National Institute of Technology Allahabad
November 8, 2012
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Outline
1 Introduction2 Motivation3 Description of First Paper
IssuesApproach Used
HDFSMapReduce Progamming Model
Example: Word Count4 Description of Second Paper
IssuesApproach Used
ArchitectureSystem Mechanism
Example5 Comparison6 Conclusion
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Introduction
MapReduce is a general-purpose programming model fordata-intensive computing.It was introduced by Google in 2004 to construct its webindex.It is also used at Yahoo, Facebook etc.It uses a parallel computing model that distributescomputational tasks to large number of nodes(approx1000-10000 nodes.)It is fault-tolerable. It can work even when 1600 nodesamong 1800 nodes fails.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Introduction
In MapReduce model, user has to write only two functions-map and reduce.Few examples that can be easily expressed asMapReduce computations:
Distributed GrepCount of URL Access FrequencyInverted IndexMining
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Motivation
Cloud Computing refers to services that are offered bycluster having 1000 to 10000 machines[6].e.g. services offered by Yahoo, Google etc.
Cloud computing deliveres computing resources as aservice. It may be -
Infrastructure as a Service (IaaS).Platform as a Service (PaaS).Software as a Service (SaaS).Storage as a Service (STaaS). etc.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Motivation cont..
Cloud Service is different from traditional hosting service infollowing ways[6] -
It is sold on demand, typically by the minute or the hour.It is elastic - a user can have as much or as little of aservice as they want at any given time.It is fully managed by provider (the consumer needsnothing but a personal computer and Internet access)
Amazon Web Services is the largest public cloud provider.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Motivation cont..
MapReduce is a programming model for large-scalecomputing[3].It uses distributed environment of the cloud to processlarge amount of data in reasonable amount of time.It was inspired by map and reduce function of FunctionalProgramming Language(like LISP, scheme, racket)[3].Map and Reduce in Racket (Functional ProgrammingLanguage)[4]:
Map:(map f list1)→ list2e.g. (map square ’(1 2 3 4 5))→ ’(1 4 9 16 25)Reduce:(foldl f init list1)→ anye.g. (foldl + 0 ’(1 2 3 4 5))→ 15
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Motivation cont..
Although, the map and reduce functions in MapReducemodel is not exactly same as in functional programming.Map and Reduce functions in MapReduce model:
Map: It process a (key, value) pair and returns a list of(intermediate key, value) pairs-
map(k1, v1)→ list(k2, v2)
Reduce: It merges all intermediate values having the sameintermediate key-
reduce(k2, list(v2))→ list(v3)
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Issues
Issues
Gaizhen Yang, "The Application of MapReduce in theCloud Computing"
It analyzes Hadoop.Hadoop is the implementation of MapReduce Model.It process data parallely in distributed manner.It divides the data into different logical blocks and processthese logical blocks in parallel on different machines and atlast combines all the results to produce the final result[1].It is fault-tolerable.One attractive feature of Hadoop is that user can write themap and reduce functions in any programming langauge.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
Approach Used
Hadoop is an open source Java framework for processinglarge amount of data on the clusters of machines[1].Hadoop is the implementation of Google’s MapReduceprogramming model.Yahoo is the biggest contributor of Hadoop[5].Hadoop has mainly two components:
Hadoop Distributed File System (HDFS)MapReduce
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
HDFS
HDFS provides support for distributed storage[1].Like traditional File System, the files can be deleted,renamed etc.HDFS has two types of nodes:
Name NodeData Node
Figure: HDFS Architecture
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
HDFS cont..
Name Node:Name Node provides the main data services.It is a process that runs on a separate machine.It stores only the meta-data of the files and directories.Programmer access files through it.For reliablity of the file system, it keeps multiple copies ofthe same file blocks.
Data Node:Data Node is a process that runs on individual machine ofthe cluster.The file blocks are stored in the local file system of thesenodes.It periodically send the meta-data of the stored blocks to theName Node.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
MapReduce Progamming Model
MapReduce is the key concept behind the Hadoop.It is a technique for dividing the work across a distributedsystem.The user has to define only two functions:
Map: It process a (key, value) pair and returns a list of(intermediate key, value) pairs-
map(k1, v1)→ list(k2, v2)
Reduce: It merges all intermediate values having the sameintermediate key-
reduce(k2, list(v2))→ list(v3)
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
MapReduce Progamming Model cont..
Execution phase of a MapReduce Application1 MapReduce library splits files into M pieces and copies
these pieces into multiple machines.2 Master picks the idle workers and assigns a map task.3 The map workers process key-value pairs of the input data
and passes each pair to the user-defined map function andproduces the intermediate key-value pairs.
4 The map worker buffers the output key-value pairs in thelocal memory. It passes these memory locations to themaster and then master forwards it to the reducer.
5 After reading the intermediate key-value pairs, the reducersorts these pairs by the intermediate key.
6 For each intermediate key, the user defined reducefunction is applied to the corresponding intermediatevalues.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
MapReduce Progamming Model cont..
7 When all map tasks and reduce tasks have beencompleted. Master gives the final output to the user.
Figure: Execution phase of a generic MapReduce Application
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Example: Word Count
Example: Word Count
The pseudo code of map and reduce function for word countproblem is -
Algorithm 3.1: MAPPER(filename, file − contents)
for each word ∈ file − contentsdo EMIT(word ,1)
Algorithm 3.2: REDUCER(word , values)
sum← 0for each value ∈ values
do sum← sum + valueEMIT(word , sum)
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Example: Word Count
Example: Word Count cont..
Figure: Word Count Execution
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Issues
Issues
Fabrizio Marozzo, Domenico Talia, Paolo Trunfioa,"P2P-MapReduce: Parallel data processing in dynamicCloud environments"
The discussed MapReduce is centralized.It can’t deal with master failure.Since the nodes joins and leaves the cloud dynamically, weneed a P2P-MapReduce model.This paper descibes an adaptive P2P-MapReduce systemthat can handle the master failure.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
Approach Used
P2P-MapReduce is a programming model in which nodesmay join and leave the cluster dynamically.The nodes act as either master or slave at a time.The master and slave interchange to each otherdynamically such that the master/slave ratio remainsconstant.To prevent the loss of computation in case of masterfailure, there are some backup masters for each masters.
The master responsible for a job J is referred as theprimary master for J.The primary master dynamically updates the job state on itsbackup nodes, which are referred as backup masters for J.When a primary master fails, its place is taken by one of itsbackup masters.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
Architecture
There are three type of nodes in P2P-MapReducearchitecture:
UserMasterSlave
The masters and slaves nodes form two logicalpeer-to-peer network M-net and S-net respectively.The composition of M-net and S-net changes dynamically.User node submits the MapReduce job to one of theavailable master nodes. The selection of master node isdone by current workload of the available master nodes.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
Architecture cont..
Master nodes perform three type of operations[2]:Management: A master node that is acting as primarymaster for one or more jobs, executes managementoperation.Recovery: A master node that is acting as backup masterfor one or more jobs, executes recovery operation.Coordination: The coordinator operation changes slavesinto masters and vice-versa, so as to keep the desiredmaster/slave ratio.
The slave executes tasks that are assigned to it by one ormore primary masters.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
Architecture cont..
For each managed jobs, primary master runs one JobManager.Backup masters runs Backup Job Manager.For each assigned tasks, slave runs one Task Manager.The task manager keeps informing to its job manager. Theinformation includes the status of the slave(ACTIVE orDEAD) and howmuch computation has been done.If a master doesn’t get the signal from a task manager,then it reschedules that assigned task on another idleslave.In addition to this condition, if a slave works slowly, thenalso the master node reschedules that assigned task onanother idle slave and consider that output which comesfirst and discards other.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Approach Used
System Mechanism
The mechanism of a generic node can be understood by UMLstate diagram[2].
Figure: Behaviour of a generic node described by an UML StateDiagram
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Example
Example
Figure: P2P-MapReduce example
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Example
Example cont..
The following recovery procedure takes place when aprimary master Node1 fails[2]:
Backup masters Node2 and Node3 detect the failure ofNode1 and starts a distributed procedure to elect the newprimary master among them.Assuming that Node3 is elected as the new primary master,Node2 continues to play the backup function and, to keepthe desired number of backup masters active, anotherbackup node is chosen by Node3.Node3 uses its local replica of the job to proceed fromwhere the Node1 fails.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Comparison between two Papers
First Paper Second PaperIssues To perform data-intensive
computation in Cloud en-vironment in reasonableamount of time.
To design a P2P MapReducesystem that can handle all thenode’s failure including Mas-ter node’s failure.
Approaches Used Simple MapReduce (pre-sented by Google) imple-mentation is used. Theimplemented version isknown as Hadoop, which isbased on the Master-SlaveModel.
Peer-to-peer architecture isused to handle all the dy-namic churns in a cluster.
Advantages Hadoop is scalable, reliableand distributed able to handleenormous amount of data. Itcan process big data in realtime.
P2P-MapReduce can man-age node churn, master fail-ures and job recovery in an ef-fective way.
Table: Comparison between two Papers.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
Conclusion
MapReduce is scalable, reliable computing model toexploids the distributed environment of the cloud.MapReduce optimizes the system performance byrescheduling the slow task on multiple slaves.P2P-MapReduce has all the property of simpleMapReduce.Since P2P-MapReduce provides fault-tolerance againstmaster failures, so it is more reliable.P2P-MapReduce prevents computation loss by keepingjob state at backup masters.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
References
Gaizhen Yang, "The Application of MapReduce in the Cloud Computing",International Symposium on Intelligence Information Processing and TrustedComputing (IPTC), October 2011, pp. 154-156, http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6103560.
Fabrizio Marozzo, Domenico Talia, Paolo Trunfioa, "P2P-MapReduce: Paralleldata processing in dynamic Cloud environments", Journal of Computer andSystem Sciences, vol. 78, Issue 5 September 212, pp.1382-1402,http://dl.acm.org/citation.cfm?id=2240494.
Jeffrey Dean and Sanjay Ghemawat, "MapReduce: simplified data processing onlarge clusters", OSDI’04 Proceedings of the 6th conference on Symposium onOpearting Systems Design & Implementation, vol. 6, 2004, pp.10-10,www.usenix.org/event/osdi04/tech/full_papers/dean/dean.pdfandhttp://dl.acm.org/citation.cfm?id=1251254.1251264..
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
References
The Racket Guide, http://docs.racket-lang.org/guide/.
Hadoop Tutorial - YDN,http://developer.yahoo.com/hadoop/tutorial/module4.html.
http://readwrite.com/2012/10/15/why-the-future-of-software-and-apps-is-serverless.
F. Marozzo, D. Talia, P. Trunfio, "A Peer-to-Peer Framework for SupportingMapReduce Applications in Dynamic Cloud Environments", In: N. Antonopoulos,L. Gillam (eds.), Cloud Computing: Principles, Systems and Applications,Springer, Chapter 7, 113-125, 2010,
IBM developer work, Using MapReduce and load balancing on the cloud, http://www.ibm.com/developerworks/cloud/library/cl-mapreduce/.
Return
Introduction Motivation Description of First Paper Description of Second Paper Comparison Conclusion References End
THANK YOU
Return