Impetus caters to diverse business needs using HPC based innovative solutions including software (Hadoop, Korus, Grids, Erlang) as well as hardware centric GPU (using CUDA) solutions. This paper explains tuning of Hadoop configuration parameters which directly affects Map-Reduce job performance under various conditions, to achieve maximum performance.
13
Abstract This paper explains tuning of Hadoop configuration parameters which directly affects Map-Reduce job performance under various conditions, to achieve maximum performance. The paper is intended for the users who are already familiar with Hadoop- HDFS and Map-Reduce. Impetus Technologies Inc. www.impetus.com October 2009 HADOOP PERFORMANCE TUNING HADOOP PERFORMANCE TUNING Abstract 1 Document Flow 2 Definitions 2 Map Reduce Workflow 2 Parameters affecting 4 Performance Other Performance Aspects 6 Conclusion 12 About Impetus 13
Transcript
1
Abstract
This paper explains tuning of Hadoop configuration parameters which
directly affects Map-Reduce job performance under various conditions,
to achieve maximum performance. The paper is intended for the users
who are already familiar with Hadoop- HDFS and Map-Reduce.
• dfs.block.size : Specifies the sizeof data blocks in which the input dataset is split
• mapred.compress.map.output: Specifies whether to compress outputof maps.
• mapred.map/reduce.tasks.speculative.execution:::::When a task (map/reduce) runs veryslowly (due to hardware degradation orsoftware mis-configuration) thanexpected. The Job Tracker runs anotherequivalent task as a backup on anothernode. This is known as speculativeexecution. The output of the task whichfinishes first is taken and the other task iskilled.
• mapred.tasktracker.map/reduce.tasks.maximum : : : : : Themaximum number of map/reduce tasksthat will be run simultaneously by a tasktracker.
• io.sort.mb : : : : : The size of in-memorybuffer (in MBs) used by map task forsorting its output.
• io.sort.factor ::::: The maximumnumber of streams to merge at oncewhen sorting files. This property is alsoused in reduce phase. It’s fairlycommon to increase this to 100.
• mapred.job.reuse.jvm.num.tasks ::::: The maximum number oftasks to run for a given job for eachJVM on a tasktracker. A value of –1indicates no limit: the same JVM maybe used for all tasks for a job.
• mapred.reduce.parallel.copies ::::: The number of threads usedto copy map outputs to the Reducer.
Map Reduce Work Flow
The diagram below explains various phases
in Map-Reduce job and the data flow across
the job.
Map Operations:Map Operations:Map Operations:Map Operations:Map Operations: Map task involves the
following actions
• Map PMap PMap PMap PMap Processing: rocessing: rocessing: rocessing: rocessing: HDFS splits the largeinput data set into smaller data blocks(64 MB by default) controlled by theproperty dfs.block.size. Datablocks are provided as an input to maptasks. The number of blocks to eachmap depends onmapred.min.split.size andmapred.max.split.size..... Ifmapred.min.split.size is less
3
W H I T E P A P E R
HADOOPPERFORMANCE TESTING
than block size andmapred.max.split.size isgreater than block size then 1 block issent to each map task. The block datais split into key value pairs based on theInput Format. The map function isinvoked for every key value pair in theinput. Output generated by mapfunction is written in a circular memorybuffer, associated with each map. Thebuffer is 100 MB by default and can becontrolled by the propertyio.sort.mb.....
• Spill:Spill:Spill:Spill:Spill: When the buffer size reaches athreshold size controlled byio.sort.spill.percent(default 0.80 or 80%), a backgroundthread starts to spill the contents todisk. While the spill takes place mapcontinues to write data to the bufferunless it is full. Spills are written inround-robin fashion to the directoriesspecified by themapred.local.dir property, in ajob-specific subdirectory. A new spill fileis created each time the memory bufferreaches to spill threshold.
• PPPPPartitioning artitioning artitioning artitioning artitioning Before writing to the diskthe background thread divides the data
into partitions (based on the partitionerused) corresponding to the Reducerwhere they will be sent.
• Sorting:Sorting:Sorting:Sorting:Sorting: In-memory sort is performedon key (based on compareTomethod of key class). The sorted outputis provided to the combiner function ifany.
• Merging:Merging:Merging:Merging:Merging: Before the map task isfinished, the spill files are merged intoa single partitioned and sorted outputfile. The configuration propertyio.sort.factor controls themaximum number of streams to mergeat once; the default is 10.
• Compression: Compression: Compression: Compression: Compression: The map output can becompressed before writing to the diskfor faster disk writing, lesser disk space,and to reduce the amount of data totransfer to the Reducer. By default theoutput is not compressed, but it is easyto enable by settingmapred.compress.map.outputto true. The compression library to useis specified bymapred.map.output.compression.codec.....
4
W H I T E P A P E R
HADOOPPERFORMANCE TESTING
Output file partitions are made available to
the Reducers over HTTP. The number of
worker threads used to serve the file
partitions is controlled by the task
tracker.http.threads property—this
setting is per tasktracker, not per map task
slot. The default of 40 may need increasing
for large clusters running large jobs.
RRRRReduce Operations:educe Operations:educe Operations:educe Operations:educe Operations: The Reducer has three
phases
1.1.1.1.1. Copy:Copy:Copy:Copy:Copy: Each map task’s output for thecorresponding Reducer is copied assoon as map task completes. Thereduce task has a small number ofcopier threads so that it can fetch mapoutputs in parallel. The default is 5threads, but can be changed by settingthemapred.reduce.parallel.copiesproperty. The map output is copied tothe reduce tasktracker’s memory bufferwhich is controlled by mapred.job.shuffle.input.buffer.percent(specifies the proportion of the heap touse for this purpose). When the in-memory buffer reaches a threshold size(controlled by mapred.job.shuffle.merge.percent), orreaches a threshold number of mapoutputs(mapred.inmem.merge.threshold),it is merged and spilled to disk. As thecopies accumulate on disk, abackground thread merges them intolarger, sorted files. This saves sometime in subsequent merging.
2.2.2.2.2. Sort: Sort: Sort: Sort: Sort: This phase should actually becalled the Merge phase as the sorting isdone at the map side. This phase startswhen all the maps have been
completed and their output has beencopied. Map outputs are mergedmaintaining their sorting order. This isdone in rounds. For example if therewere 40 map outputs and the mergefactor was 10 (the default, controlled bythe io.sort.factor property, justlike in the map’s merge) then therewould be 4 rounds. In first round 4 fileswill be merged and in remaining 3rounds 10 files are merged. The lastbatch of files is not merged and directlygiven to the reduce phase.
3.3.3.3.3. RRRRReduce: educe: educe: educe: educe: During reduce phase thereduce function is invoked for each keyin the sorted output. The output of thisphase is written directly to the outputfilesystem, typically HDFS.
o Small cluster and large data set:Small cluster and large data set:Small cluster and large data set:Small cluster and large data set:Small cluster and large data set:default block size will create a largenumber of map tasks.
e.g. Input data size = 160 GB anddfs.block.size = 64 MB thenthe minimum no. of maps=(160*1024)/64 = 2560 maps.
If dfs.block.size = 128 MBminimum no. of maps=(160*1024)/128 = 1280 maps.
If dfs.block.size = 256 MBminimum no. of maps=(160*1024)/256 = 640 maps.
In a small cluster (6-7 nodes) themap task creation overhead isconsiderable. So dfs.block.sizeshould be large in this case but small
5
W H I T E P A P E R
HADOOPPERFORMANCE TESTING
enough to utilize all the clusterresources.
o The block size should be setaccording to size of the cluster, maptask complexity, map task capacity ofcluster and average size of inputfiles. If the map contains thecomputation such that one datablock is taking much more time thanthe other block, then the dfs blocksize should be lesser.
mapred.compress.map.output: Map
Output Compression
• Default:Default:Default:Default:Default: False
• PPPPPros: ros: ros: ros: ros: Faster disk write, saves diskspace, less time in data transfer (fromMappers to Reducers).
• Cons: Cons: Cons: Cons: Cons: Overhead in compression atMappers and decompression atReducers.
• Suggestions:Suggestions:Suggestions:Suggestions:Suggestions: For large cluster andlarge jobs this property should be settrue. The compression codec can alsobe set through the property mapred.map.output.compression.codec(Default is org.apache.hadoop.io.compress.DefaultCodec).
• PPPPPros: ros: ros: ros: ros: Reduces the job time if the taskprogress is slow due to memoryunavailability, hardware degradation.
• Cons: Cons: Cons: Cons: Cons: Increases the job time if the taskprogress is slow due to complex and
large calculations. On a busy clusterspeculative execution can reduceoverall throughput, since redundanttasks are being executed in an attemptto bring down the execution time for asingle job.
• Suggestions:Suggestions:Suggestions:Suggestions:Suggestions: In large jobs whereaverage task completion time issignificant (> 1 hr) due to complex andlarge calculations and high throughputis required the speculative executionshould be set to false.
o For large jobs (the jobs in whichmap output is very large), this valueshould be increased keeping in mindthat it will increase the memoryrequired by each map task. So theincrement in this value should beaccording to the available memoryat the node.
o Greater the value of io.sort.mb,lesser will be the spills to the disk,saving write to the disk.
o For large jobs (the jobs in whichmap output is very large andnumber of maps are also large)which have large number of spills todisk, value of this property should beincreased.
o Increment in io.sort.factor,benefits in merging at Reducers sincethe last batch of streams (equal toio.sort.factor) are sent to thereduce function without merging,thus saving time in merging.
mapred.job.reuse.jvm.num.tasks:
Reuse single JVM
• Default: Default: Default: Default: Default: 1
• Suggestions: Suggestions: Suggestions: Suggestions: Suggestions: The overhead of JVMcreation for each task is around 1second. So for the tasks which live forseconds or a few minutes and havelengthy initialization, this value can beincreased to gain performance.
mapred.reduce.parallel.copies:
Threads for parallel copy at Reducer
• Default: Default: Default: Default: Default: 5
• Description: Description: Description: Description: Description: The number of threadsused to copy map outputs to theReducer.
• Suggestions: Suggestions: Suggestions: Suggestions: Suggestions: For large jobs (the jobs inwhich map output is very large), valueof this property can be increasedkeeping in mind that it will increase thetotal CPU usage.
Other Performance Aspects
TTTTTemporary space: emporary space: emporary space: emporary space: emporary space: Jobs which generate
PPPPProblem Statement:roblem Statement:roblem Statement:roblem Statement:roblem Statement: Processing more than
160 GB of data logs based on some
business logic, using around 1.2 GB of
metadata (complex hierarchical look up/
master data). Processing involves complex
business logic, grouping based on different
fields and sorting. It also involves
processing metadata from raw metadata
files.
Challenges:
1. Metadata is not actually metadatasince it is updated while processingbased on some field.
2. Processing generates hugeintermediate data (~ 15 TB).
3. Large number of records (> 170million) in a few aggregation groups.
4. Metadata (~1.2 GB) should remainin-memory while processing.
5. Providing large metadata to differentparallel jobs and tasks.
6. Solution to be run on small cluster of 7nodes.
7. Multiple outputs were required fromsingle input files.
Cluster Specifications:
Datanodes+tasktrackers = 6machines
Namenode+jobtracker = 1 machine
Each machine has 8GB main memory,
8 core CPU, 5 TB Hard drive
Cloudera Hadoop 0.18.3
Approaches: Approaches: Approaches: Approaches: Approaches: The problem was divided into
5 Map Reduce jobs- Metadata job, Input
processing job for parsing the input files
and generating 2 outputs, intermediate
output 1 processing job, intermediate
9
W H I T E P A P E R
HADOOPPERFORMANCE TESTING
output 2 processing job and an additional
downstream aggregation job.
Metadata PMetadata PMetadata PMetadata PMetadata Processing:rocessing:rocessing:rocessing:rocessing: The first step was to
process metadata from raw metadata files
using some business logic. The metadata
was in the form of in memory hash maps.
Here the challenge (5) was how to provide
this in memory data to different jobs and
tasks in parallel environment since Hadoop
doesn’t have global (shared) storage
among different tasks. The solution arrived
at was to serialize all the hash maps and
store into the files (~ 390 MB) then add
these files to distributed cachedistributed cachedistributed cachedistributed cachedistributed cache (A cache
provided by Hadoop, which copies all the
resource added into it, to the local file
system of each node) and de-serialize it in
all the tasks.
o Approach 1: Approach 1: Approach 1: Approach 1: Approach 1: Due to the challenge 1, itwas decided that whichever jobupdates metadata should be processedsequentially i.e. only 1 map and 1reduce task should run. The mainprocessing job had the business logicas part of Maps.
Feasibility: This approach worked onlyfor small input data set (< 500 MB),since the memory requirements werehigh due to the reason thataggregation records (key,value) raninto 170 million+ records in somecases. Overall this approach wasunfeasible for the target input data set
o Approach 2: Approach 2: Approach 2: Approach 2: Approach 2: As a solution forchallenge 1, all the processing wasmoved to the Reducers and the mapoutput was partitioned on the fields onthe basis of which, the metadata wasupdated. Hadoop default hashpartitioner was used and the mapoutput key’s hash code method waswritten accordingly. Now all the jobscould run in parallel.
Feasibilty: This approach couldprocess more data then the previousapproach but was still not feasible toprocess the target input data set.
o Approach 3:Approach 3:Approach 3:Approach 3:Approach 3: Two jobs (inputprocessing and intermediate output 1)were merged together (since one jobhad all the processing on Mapper andother on Reducer) to save time to extraread and write (>320 GB) to HDFS.Now this combined job was generatinghuge intermediate data ~ 15 TB(challenge 2) which cause the job tofail with error
Exception in thread “main”org.apache.hadoop.fs.FSError:java.io.IOException: Nospace left on device atorg.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFileSystem.java:150)
The solution to the above error was toincrease the space provided to themapred.local.dir directories sothat 15 TB data can be stored. Inaddition to the above solution thepropertymapred.compress.map.outputwas set to true.
Now came the challenge 3, thegrouping on 1 field causes ~170million records in a few groups and thebusiness logic was such that the
10
W H I T E P A P E R
HADOOPPERFORMANCE TESTING
intermediate data generated byprocessing each group should remainin memory unless the new groupcomes, causing memory shortage andultimately the job failure. The solutionto this was tweaking the business logicsuch that the grouping can be done onmore than one fields thereby reducingthe number of records in a group.
Another optimization to overcome theeffect of challenge 6 was to increasedfs.block.size to 256 MB since itsdefault value (64MB) was creating~5000 maps. So in a small cluster of7 nodes the map creation overheadwas significant. After the block sizechange, 631 maps were being createdfor input data.
Feasibility: This approach was feasiblebut the estimated time to processtarget input data set was more than200 hrs.
o Approach 4:Approach 4:Approach 4:Approach 4:Approach 4: Another solution to thechallenge 4 was to use memcached formetadata, instead of keeping it inmemory, so that number of maps canbe increased. First the spy memcachedclient was used but it had some dataleakages and didn’t work. FinallyDanga memcached client was usedand it worked well. Throughput usingmemcached was around 10000transactions per second.
Here are some statistics of the jobrunning with memcached
ConclusionConclusionConclusionConclusionConclusion - MemcachedMemcachedMemcachedMemcachedMemcached servers wereshowing a thruputthruputthruputthruputthruput of around 10000fetches per second (150000 hits persecond across 15 nodes). However,this was not sufficient to meet therequirements for faster processing ormatch up with in memory hash maplookups.
o Approach 5: Approach 5: Approach 5: Approach 5: Approach 5: This approach uses thefact that Reducer would start(processing data) after copy andreduce by which time; MappersMappersMappersMappersMappers wouldhave shut down and released thememory (Reducer Lazy Initialization).
Feasibility: Time Taken ~42 hrs42 hrs42 hrs42 hrs42 hrs
HDFS data processing rate ~8 GB perhour
Maps finished = 19 hrs 37 mins
No. of Mappers per node = 5
No. of Mappers= 5*6 = 30
No. of Reducers= 2*6 = 12
HDFS chunk size = 256 MB
Io.sort.mb = 300
Io.sort.factor = 75
Memory usage per node
Per node: RAM- ~8 GB, SWAP- 4-6GB
Per Mapper memory ~ 1.1-1.8GB
Per Reducer memory ~ 50 MB(during copy) + 1.1-2 GB (duringsort and reduce)
Total memory usage in Mapperprocessing ~ 1.5 GB *5 + 50 MB*2 + 1 +1 (tasktracker anddatanode) ~ 9.6GB
o Approach 6:Approach 6:Approach 6:Approach 6:Approach 6: Approach 5 was runagain with the following configurations
Total memory usage in Mapperprocessing ~ 1.8 GB *4 + 50 MB *5+ 1 +1 (tasktracker anddatanode) ~ 9.5GB
Feasibility: Time taken ~ 39hrs39hrs39hrs39hrs39hrs
Approach 6 gave maximumperformance and took minimum timein the given cluster specifications. Only4 Map jobs per node could be run asuse of any other additional Map wascausing excessive swap memory whichwas a major bottleneck for fasterprocessing.
12
W H I T E P A P E R
HADOOPPERFORMANCE TESTING
o Approach 7:Approach 7:Approach 7:Approach 7:Approach 7: Approach 6 was runagain but with Tokyo cabinet (a BerkleyDB like file storage system) formetadata handling to overcomeexcessive RAM usage to storemetadata with the followingconfigurations
