Session9_MR2_Sep25

8/11/2019 Session9_MR2_Sep25

http://slidepdf.com/reader/full/session9mr2sep25 1/28

Inspire…Educate…Transform.

The best place for students to learn Applied Engineering http://www.insofe.ed

Dr. Sreerama KV Murthy

September 25, 2013

Engineering Big Data:Online Batch

Session 9: Map Reduce 2

CEO, Teqnium Consultancy Services



The best place for students to learn Applied Engineering 2 http://www.insofe.ed

Refresher: What is MapReduce?

MapReduce is a programming model Google has used

successfully is processing its “big-data” sets (~ 20000 petabytes per day)

Users specify the computation in terms of a map and a reduce

function

Underlying runtime system automatically parallelizes the computationacross large-scale clusters of machines, and

Underlying system also handles machine failures, efficientcommunications, and performance issues.

Reference: Dean, J. and Ghemawat, S. 2008. MapReduce: simplified

data processing on large clusters. Communication of ACM 51, 1 (Jan2008), 107-113.

CCSCNE 2009 Palttsburg, April 24 2009. B.Ramamurthy & K.Madurai




The Five MapReduce Daemons

1. NameNode

• Holds the metadata for HDFS

2. Secondary NameNode

• Performs housekeeping functions for the NameNode. It is not a backup or

hot standby for the NameNode.

3. DataNode

• Stores actual HDFS data blocks

4. JobTracker • Manages MapReduce jobs, distributes individual tasks to machines, etc

5. TaskTracker

• Instantiates and monitors individual Map and Reduce tasks

“Master Nodes” in the cluster run one of the blue daemons above.

“Slave Nodes” run both of the non-blue daemons.

Each daemon runs in its own Java virtual machine.




Five Daemons of MapReduce.. contd.


http://slidepdf.com/reader/full/session9mr2sep25 5/28The best place for students to learn Applied Engineering 5 http://www.insofe.ed

YARN (MR2)



CDH-3 Map Reduce Daemons



MapReduce NextGen aka YARN aka MRv2

• Divides the two major functions of the JobTracker - resource

management and job life-cycle management - into separatecomponents

• Released in Hadoop-0.23

• The new ResourceManager manages the global assignment ofcompute resources to applications.

• The ResourceManager has two main components: Scheduler andApplicationsManager.

• The Scheduler is responsible for allocating resources to variousrunning applications subject to constraints of capacities, queues etc.



MRv2 aka YARN: JobTracker Redefined



• Scheduler performs no tracking of status for the application, andoffers no guarantees about restarting failed tasks.

• The per-application ApplicationMaster manages the application’sscheduling and coordination

• The per-machine NodeManager daemon manages the userprocesses on that machine.

• An application is either a single MR job or a DAG of such jobs.

• The ApplicationMaster negotiates resources from theResourceManager and works with the NodeManager(s) to executeand monitor tasks.

• CDH4 continues to support the original MapReduce framework (i.e.the JobTracker and TaskTrackers). The old framework is referred toas MRv1.

YARN aka MRv2 (contd.)





A COUPLE OF USE CASESMap-Reduce



Yahoo: Running Production WebMap

• Search needs a graph of the “known” web

– Invert edges, compute link text, whole graph heuristics

• Periodic batch job using Map/Reduce

– Uses a chain of ~100 map/reduce jobs

• Scale

– 1 trillion edges in graph

– Largest shuffle is 450 TB

– Final output is 300 TB compressed

– Runs on 10,000 cores

– Raw disk used 5 PB

• Written mostly using Hadoop’s C++ interface



Yahoo Research Clusters

• Mostly data mining/machine learning jobs

• Most research jobs are not Java:

–42% Streaming

• Uses Unix text processing to define map and

reduce–28% Pig

• Higher level dataflow scripting language

–28% Java

–2% C++






Terabyte Sort Benchmark

• Started by Jim Gray at Microsoft in 1998• Sorting 10 billion 100 byte records

• Hadoop won the general category in 209 seconds– 910 nodes

– 2 quad-core Xeons @ 2.0Ghz / node

– 4 SATA disks / node

– 8 GB ram / node– 1 gb ethernet / node

– 40 nodes / rack

– 8 gb ethernet uplink / rack

• Previous records was 297 seconds

• Only hard parts were:– Getting a total order

– Converting the data generator to map/reduce




NOW….THE

SHAKE-UP QUIZ !!




KEY-VALUE PAIRS






Keys and Values




MapReduce – In more detail




MR: Logical Execution

k k k k k kv v v v




mapmap map map

Shuffle and Sort: aggregate values by keys

reduce reduce reduce

k1 k2 k3 k4 k5 k6v1 v2 v3 v4 v5

v6

ba 1 2 c c3 6 a c5 2 b c7 8

a 1 5 b 2 7 c 2 3 6 8

r 1 s1 r 2 s2 r 3 s3




SIMPLE MAPPERS & REDUCERS

M




Mappers

• Mappers run on nodes which hold their portion of the data locally,to avoid network traffic

• Multiple Mappers run in parallel, each processing a portion of theinput data

• Mapper reads data in the form of key/value pairs

– Mapper may use, or completely ignore, the input key.

– E.g., a standard pattern is to read a line of a file at a time. Key then isthe byte offset into the file at which the line starts. Value is thecontents of the line itself. Typically the key is considered irrelevant .

• It outputs zero or more key/value pairs

– let map(k, v) = emit(k.toUpper(), v.toUpper())

– ('foo', 'bar') -> ('FOO', 'BAR')

Oth E l E l d




Others Examples: Explode mapper

E l Filt




Example: Filter mapper

E l Ch i K




Example: Changing Keyspaces




International School of Engineering

Plot No 63/A, 1st Floor, Road No 13, Film Nagar, Jubilee Hills, Hyderabad - 500033

For Individuals: +91-9502334561/63

For Corporates: +91-9618483483

Web: http://www.insofe.edu.in

Facebook: https://www.facebook.com/Insofe

Twitter: https://twitter.com/INSOFEedu

YouTube: http://www.youtube.com/InsofeVideos

SlideShare: http://www.slideshare.net/INSOFE

This presentation may contain references to findings of various reports available in the public domain. INSOFE makes no representation as to their accuracy or that the organization

subscribes to those findings.

http://www.inse.edu.in/

https://www.facebook.com/Insofe

https://twitter.com/INSOFEedu

http://www.youtube.com/InsofeVideos

http://www.slideshare.net/INSOFE

http://www.slideshare.net/INSOFE

http://www.youtube.com/InsofeVideos

https://twitter.com/INSOFEedu

https://www.facebook.com/Insofe

http://www.inse.edu.in/

Date post:	02-Jun-2018
Category:	Documents
Upload:	mbscribd2011
View:	213 times
Download:	0 times

Session9_MR2_Sep25

Documents