A Data Skew Oriented Reduce Placement Algorithm Based on...

2168-7161 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for moreinformation.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TCC.2016.2607738,IEEE Transactions on Cloud Computing

1

A Data Skew Oriented Reduce PlacementAlgorithm Based on Sampling

Zhuo Tang, Wen Ma, Kenli Li, Member, IEEE , and Keqin Li, Fellow, IEEE

Abstract—For frequent disk I/O and large data transmissions among different racks and physical nodes, intermediate datacommunication has become the most important performance bottle-neck in most running Hadoop systems. This paper proposesa reduce placement algorithm called CORP to schedule related map and reduce tasks on the near nodes of clusters or racks fordata locality. Because the number of keys cannot be counted until the input data are processed by map tasks, this paper appliesa reservoir algorithm for sampling the input data, which can bring the distribution of keys/values closer to the overall situation oforiginal data. Based on the distribution matrix of the intermediate results in each partition, by calculating the distance and costmatrices among the cross node communication, the related map and reduce tasks can be scheduled to relatively nearby physicalnodes for data locality. We implement CORP in Hadoop 2.4.0 and evaluate its performance using three widely used benchmarks:Sort, Grep, and Join. In these experiments, an evaluation model is proposed for selecting the appropriate sample rates, whichcan comprehensively consider the importance of cost, effect, and variance in sampling. Experimental results show that CORPcan not only improve the balance of reduces tasks effectively but also decreases the job execution time for the lower inner datacommunication. Compared with some other reduce scheduling algorithms, the average data transmission of the entire systemon the core switch has been reduced substantially.

Index Terms—data sampling, data skew, inner communication, MapReduce, reduce placement

F

1 INTRODUCTION1.1 motivation

W ITH the rapid development of the Internet andthe exponentially increasing size of data, da-

ta parallel programming models have been widelyused in processing terabyte- and petabyte-intensivedistributed data, such as MapReduce [1], MPI [2],and OpenMP [3]. In particular, Hadoop [4] is anopen source implementation of MapReduce and iscurrently enjoying wide popularity; however, it stillhas room for improvement, such as intermediate datafault tolerance [5], data skewness [6], and localizeddata [7]. This paper focuses on data-locality and inter-node network traffic in Hadoop, which are criticalfactors in the high performance of the MapReduceframework.

In the Hadoop framework, because map tasks al-ways output the intermediate data in the local n-odes, the data should be transmitted from the mapnodes to corresponding reduce nodes, which is an all-to-all communication model. The frequent disk I/Oand large data transmissions have become the mostsignificant performance bottle-neck in most runningHadoop systems, which may saturate the top-of-rackswitch and inflate job execution time [8]. Cross-rack

• The authors are with the College of Information Science and Engi-neering, Hunan University, and National Supercomputing Center inChangsha, Hunan, China, 410082.E-mail: [email protected], [email protected], [email protected].

• Keqin Li is also with the Department of Computer Science, StateUniversity of New York, New Paltz, New York 12561, USA.

• Zhuo Tang is the author for correspondence.

communication occurs if a mapper and a reducerreside in different racks, which is very common inthe environment of current data centers [9]. Due tothe limitation of the switches in clusters, the overallperformance of a system is often not satisfactory whenworking on large data-sets [10].

For map tasks, which always start at the node withcurrent input data, to mitigate network traffic, aneffective method is to place reduce tasks near thephysical nodes on which map tasks generate inter-mediate data used as the reduce input [11]. Becausethe intermediate key distribution is the determiningfactor for the input data distribution of reduce tasks, ifthe intermediate data from map tasks are distributedto reduce tasks uniformly, reduce locality and taskplacement are unable to optimize the all-to-all com-munication in Hadoop. Luckily, data skew is univer-sally existent in all input data. Current research provesthat moving tasks is more efficient than moving datain the Hadoop distributed environment [12], wheredata skews are widespread (some key values aresignificantly more frequent than others). These studiesallow us to solve the cross-rack/node communicationproblem through reduce task placement.

Most versions of Hadoop usually employ statichash functions to partition the intermediate data. Thisworks well only when the data are uniformly dis-tributed, but performs poorly when the intermediateresults of the input are skewed. Fig. 1 illustrates thedifferent amount of input data for each reduce taskwhen running the ”WordCount” benchmark using10GB of data. For reduce tasks, partitioning skewwill cause shuffle skew, in which some reduce tasks



2

10 20 300.0

0.5

1.0

1.5

Dat

a Pr

oces

sed(

GB

)

Reduce ID

The Data Average

Ave

rage

Fig. 1: Partitioning skew in reduce tasks

will receive more data than others. The shuffle skewproblem would degrade the performance in Hadoop,because a job may be delayed by a reduce task fetch-ing large input data.

To improve system performance in this situation,many studies have focused on the data skew miti-gation and tasks load balance at present. Typically,research has addressed the problems of how to im-prove system performance by efficiently partitioningthe intermediate keys to guarantee fair distribution ofthe inputs for reducers [13], improving data localityby direct task placement [7], [14], [15], or indirect taskplacement based on virtual machine immigration [16].Others have attempted to improve resource utilizationthrough speculation execution [5], [17] or task place-ment [15].

With these existing studies, the starting point of ourwork is not to solve the problems caused by dataskew, but to provide a fine-grained detection methodfor the skewed intermediate data distribution to op-timize the inner cross-racks/nodes communicationthrough task placement. In this processing, throughthe calculation of the cost matrix, we consider boththe data locality and the load balance.

In the Hadoop architecture, data locality and loadbalance are not contradictory goals, and there is no di-rect correlation between these two aspects. At present,many studies are attempting to optimize them syn-chronously. For example, Ioan et al. proposed a data-aware work stealing technique that is able to achievegood load balancing, and yet still tries to best exploitdata-locality [18]. In their implementations, tasks arelaunched locally, but they could be migrated amongschedulers for balancing loads through work stealing.

We draw many inspirations from these works.First,in the processing of a Hadoop job, the map taskswill begin at the node with its necessary input data.Although data locality is very important for maptasks, data unbalance can also damage the localitybecause individual node managers cannot afford ex-cessive map tasks. Second, data skew will cause theimbalance among different reducers because of thekeys dispatching based on hash, but it is necessaryto consider the problem of how to place the reducetasks based only on this imbalance condition: how toimplement the locality by starting the reduce tasks on

the nodes that are the source nodes of their input data.

1.2 Our ContributionsThis paper proposes a communication-oriented re-

duce placement (CORP) method to reduce all-to-allcommunications between mappers and reducers, andits basic idea is to place related map and reduce taskson the near nodes of clusters or racks. Because dataskew is difficult to solve if the input distribution isunknown, a normal thought is to examine the databefore determining the partition. In a real applica-tion, the intermediate outputs can be monitored andcounted only after a job begins running, but it ismeaningless to obtain the key value distribution afterprocessing all input data.

To address this problem, this paper provides a dy-namic range partition method that conducts a prerunsample of the input before the real job. By integratingsampling into a small percentage of the map tasks,this paper prioritizes the execution of sampling tasksover the normal map tasks to achieve the distributionstatistics. The main contributions of this paper aresummarized below.• We apply the reservoir algorithm to imple-

ment the sampling for input data, and propose anevaluation model to select the appropriate samplerate. This model can comprehensively consider theimportance of cost, effect, and variance in sampling.• We propose a novel reduce placement algo-

rithm based on data distribution, which can schedulethe related map and reduce tasks on the near nodesfor data locality. This algorithm can reduce the all-to-all communication among inner Hadoop clusters.• We implement CORP in Hadoop 2.4.0 and e-

valuate its performance for some of the most commonbenchmarks. Experiment results show that CORP re-duce the data transmission on core switch significant-ly compared with the default hash mechanism.

The rest of the paper is organized as follows. Section2 surveys related works on reducer placement anddata skew. Section 3 introduces the overall systemframework. Section 4 proposes the data samplingalgorithm of the MapReduce framework. Section 5proposes the reduce placement algorithm. The per-formance evaluation is given in Section 6. Section 7concludes the paper.

2 RELATED WORKS

To optimize the performance in the Hadoop frame-work, many algorithms and models for reduce taskscheduling have been proposed in recent years.Through analysis of the current MapReduce schedul-ing mechanism, our early work illustrated the rea-sons for system slot resource wasting, which resultsin starvation of reduce tasks. We proposed a self-adaptive reduce task scheduling model (SARS) forthe start time of reduce tasks [19]. Without the space



3

deployment ability, SARS just determine the start timepoint of each reduce task dynamically according tothe predicted completion time and current size of themap output. That is, SARS cannot improve the datalocality and lighten the network loads in the Hadoopcluster.

Data skew is not a new problem specific to MapRe-duce. For the typically skewed distribution of inter-mediate data in Hadoop, we must face many realworld applications exhibiting significant data skew,including scientific applications [20], [17]; distributeddatabase operations such as join, grouping and ag-gregation [21]; search engine applications (Page Rank,Inverted Index, etc.) and some simple applications(sort, grep, etc.) [4]. Methods by which to handle data-skew effects have been studied previously in paralleldatabase researches[22], [23], [24], but there is still noeffective prediction model for the distribution of theintermediate keys.

The following studies are more similar to ourworks.Ibrahim et al. [13] developed a novel algo-rithm named LEEN for locality-aware and fairness-aware key partitioning in MapReduce. LEEN em-braces asynchronous map and reduce schemes. Allbuffered intermediate keys are partitioned accordingto their frequencies and the fairness of the expecteddata distribution after the shuffle phase. However, itlacks preprocessing to estimate the data distributioneffectively, so this proposed mechanism may incursignificant time cost to scan the table of keys frequen-cies, which is generated after map tasks. This aspectof time costs is not considered in their experiments.

SkewTune is a partition algorithm proposed byKwon et al. [17], which can remit the skew of inter-mediate data by redistricting the larger partition ofoutput data from map tasks. This method is similar tothe improvement of the Range algorithm [25], whichis also used as a comparable algorithm in our experi-ments. SkewTune cannot pinpoint or split exact largekeys because it does not sample any key frequen-cy information. Therefore, as long as large keys aregathered and processed, the system cannot rearrangethem. Its reduce skew mitigation cannot improve copyand sort phases, which cause performance bottleneckfor some applications.

The main advantage of the works by Gufler et al.[26] is TopCluster, a proposed distributed monitoringsystem for capturing data skew in MapReduce sys-tems, that can provide the cost estimation for eachintermediate partition from map tasks. Hence, thepartitions are distributed to the reducers such thatthe work load per reducer is balanced. Because theconcern of this work is partition processing ratherthan tasks placement, the problem of data localitycannot be solved well under this model, Similar toSkewTune [17].

Tan et al. [15] formulated a stochastic optimizationframework to improve the data locality for reduce

tasks, with the optimal placement policy exhibitinga threshold-based structure. Their other work im-plemented a resource-aware scheduler for Hadoop[27] that couples the progress of map tasks and re-duce tasks, utilizing wait scheduling for reduce tasksand random peeking scheduling for map tasks tojointly optimize the task placement. These excellentworks are based on improving the utility of slotsresources, which can improve the data locality butwithout enough consideration of the load balance.In addition, it is also difficult to apply in currentversions of Hadoop with Yarn resource managementcomponents.

Chen et al. [6] presented LIBRA, a lightweightstrategy to solve the data skew problem for reduce-side applications in MapReduce. LIBRA estimatesthe intermediate data distribution by sampling thepartial map tasks, and uses an innovative approachto balance the load among the reduce tasks, whichsupports the split of large keys. Their solutions canreduce the overhead while estimating the reducersworkload, but these solutions still have to wait forthe completion of all the map tasks for whole inputdata. In addition, because data sampling is always anadditional step of work for running jobs, it inevitablyincurs extra running times and degrades the overallsystem performance.

In conclusion, there are still some problems that arenot solved perfectly in these previous studies: (1) howto detect the intermediate data distribution efficientlyfull scanning in Hadoop job processing seems a bitineffective; and (2) how to implement fine-grainedcontrol for the task placement: it should be modelledand quantified by an accurate cost evaluation modelin the runtime environment.

3 SYSTEM OVERVIEW

Our standpoint is that if map tasks and correspond-ing reduce tasks are placed close to each other (on thesame server, same rack, etc.), the system would costless for the same amount of traffic relative to a casewith the reduce tasks located far from the node. Fur-thermore, in addition to improving the performanceof an application, minimizing the communication costwill also reduce the network overhead of the under-lying infrastructure by moving traffic from bottlenecklinks to high-bandwidth links.

Fig. 2 shows the overall execution produce, whichis composed of two separate jobs. First, the originalinput data are sampled to estimate the source ofkey/value tuples for each reducer by nodes. Theoutput of this phase is a matrix to record the size andkey/value distribution of current input data, whichcan be transferred to a cost matrix for reduce taskplacement. Before the working job is run, the reducetask placement should be finished according to thismatrix. That way, most data handled by a reduce



4

task handles be localized as much as possible, thussaving traffic cost and improving the performance ofthe reduce tasks. The main steps are as follows.

Data Sampling: Input data are loaded into a file orfiles in a distributed file system (DFS) where each fileis partitioned into smaller chunks, called input splits.Each split is assigned to a map task. Map tasks processsplits, and produce intermediate outputs which areusually partitioned or hashed to one or many reducetasks. Before a MapReduce computation begins witha map phase, in which each input split is processed inparallel, a random sample of the required size will beproduced. The splits of samples are submitted to theauditor group, while the master and map tasks waitfor the results of the auditor.

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

� ��

��

��

��

��

��!��"��#�$��

Fig. 2: The framework of CORP

Reduce Task Placement: The results of samplingwill determine the placement of reduce tasks. Fig. 3briefly shows a typical example. For 80% key/valuepairs, reduce task R1 comes from map task M2, andthe remaining intermediate results are from map taskM1. Hence, the most appropriate position to start thetask of reduce task R1 is the node on which maptask M2 is running. Analogously, to obtain better datalocality and save the inner communication amongnodes, it is better to launch reduce task R2 on thenode of map task M1.

{<k,v>}

{<k,v>}

Hash(k) Mod R

Map1:20%

Map2:80%

Map1:40%

Map2:60%

Fig. 3: The intermediate results distribution inreduce tasks

4 DATA SKEW AND DATA SAMPLING INMAPREDUCE FRAMEWORK4.1 Data Skew Model

In this model, to quantify the data received bya special reduce task, some initial and intermediate

results with their relationships can be formalized asTable. 1.

TABLE 1: Variable Declaration

n, 0 ≤ l < n n: node number; l: one node;p, 0 ≤ j < p p: reducer number; j: one reducer;m, 0 ≤ i < m m: mapper number; i: one mapper;Cσ,l

i,j key/value numbers from the mapper Mi innode Nl received by jth reducer;

RC(j) number of key/value pairs processed by re-ducer j;

meanσ average number of key/value tuples of allrunning reduce tasks;

std the standard deviation for the current loadingof reducer;

FoS an indicator to measure the load balance ofreducer.

C = Cσ,li,j is a three-dimensional matrix of m × p ×

n that defines the distribution of intermediate resultsin each partition. Cσ,l

i,j denotes the number key/valuetuples processed by the jth reducer from the ith maptask within the lth node. And Cσ,l

i,j = k means thatk pairs of keys/values from map task Mi in node Nl

are currently allocated to reduce task j. For a partitionthat represents the tuple set processed by the samereducer, the number of partitions is treated equallyto the reducer amount. In this model, n denotes thenumber of nodes, p denotes the number of reducers,and m denotes the number of map tasks. Hence, 0 ≤l < n, 0 ≤ j < p, and 0 ≤ i < m.

Under normal conditions, the key number of o-riginal input data follows a Zipf distributions [28].Parameter σ is used to denote the degree of theskew, which is usually assigned from 0.1 to 1.2. Fora specific input dataset, parameter σ is a constant.A larger value indicates heavier skew, and it alsodetermines the distribution of Ci,j . In this model, thenumber of key/value pairs processed by the reducerj is denoted as RC(j). Without loss of generality, thevalue of RC(j) with a skew degree could be definedas follows:

RC(j, σ) =n−1∑l=0

Cσ,lj =

n−1∑l=0

m−1∑i=0

Cσ,li,j (1)

On this basis, we can calculate the average numberof key/value tuples of all running reduce tasks as Eq.(2):

meanσ =

p−1∑j=0

RC(j, σ)

p=

p−1∑j=0

n−1∑l=0

m−1∑i=0

Cσ,li,j

p

(2)

in which parameter p is the number of reduce tasks.Naturally, the intermediate data processed by a

reduce task can be considered as skew using standarddeviation. As shown in Eq. (3):

|RC(j, σ)−meanσ| > std (3)



5

std is the standard deviation of the number ofkey/value tuples for all reduce tasks, which can beused to measure the overall load balancing level ofreducers.

Further, we can evaluate the difference between theaverage intermediate results of all reduce tasks andthe number of key/value pairs belonging to the jth

reducer as Eq. (4):

(RC(j, σ)−meanσ) =

n−1∑l=0

m−1∑i=0

Cσ,li,j

−

p−1∑j=0

n−1∑l=0

m−1∑i=0

Cσ,li,j

p

(4)

The value of this standard deviation for all inter-mediate results in reduce tasks can be calculated byEq. (5):

std[(RC(j, σ)] =

√√√√√p−1∑j=0

(RC(j, σ)−meanσ)2

p

=

√√√√√√√√p−1∑j=0

n−1∑l=0

m−1∑i=0

Cσ,li,j −

p−1∑j=0

n−1∑l=0

m−1∑i=0

Cσ,li,j

p

2

p

(5)

In this case, when a reduce task is load balanced,|RC(j, σ) − meanσ| < std is always satisfied. As aresult, when the number of key/value tuples assignedto reducer j is larger than the value of the mean, thejth reducer will be taken as a skew task even althoughit is running normally.

To measure the data skew degree of all reduce tasks,this paper uses the indicator FoS (Factor of Skew) toquantize data skew and load balancing:

FoS = std[(RC(j, σ)]/meanσ (6)

The smaller the value of FoS, the better the loadbalancing and the lower data skew that will be ob-tained.

4.2 Data Sampling AlgorithmTo ascertain the distribution of the intermediate

data is the only way to develop a reduce placemen-t strategy. Because the number of keys cannot becounted until the input data are processed by maptasks, calculating the optimal solution to the aboveproblem is unrealistic, and the cost of pre-scanningthe whole dataset would likely be unacceptable whenthe amount of data is huge. Therefore, we present adistributed approximation algorithm by sampling andestimation.

In most running Hadoop systems, sampling of theinput data can be achieved by using the class: org.

apache. hadoop. mapred. lib. InputSampler. This classimplements the SplitSampler method which samplesrecords from the first S partitions. The original releaseis a random sampling of data convenient way. Read-ing a record will be added to the sample set in eachpartition as long as the current number of samplesis less than samples needed, by means of loopingthrough all records in this partition. Our sample s-trategies are also run by invoking this InputSamplerclass in the underlying implementation. To improvethe initial random selection strategy, this paper over-loads the SplitSampler method and proposes a moreefficient selection model for sample data based onreservoir sampling.

Conventional uniform sampling will inevitably re-sult in a certain number of multi-duplicated samples.Because all random number generators in the Javaand Scala languages are simply pseudorandom func-tions, for large-scale data, especially with increasingsampling space, they cannot guarantee that all sampledata are completely randomized. From [29], the mainprocess of reservoir sampling is to save k precedingelements first (k is the sample number and also thesize of the reservoir) and then randomly replace o-riginal selected elements in the reservoir using a newelement that is selected from outside the reservoir ina different probability. The final k sample data willbe generated after finishing the traversal for currentinput data. Compared with pseudorandom functions,reservoir sampling can ensure randomness, especiallywhen taking the data from some sequence flows, andit is ideal for reading the input data from large textsline by line in the Hadoop/Spark framework. Com-pared with conventional uniform sampling, reservoirsampling can ensure that the key distributions arecloser to the whole situation in the original data.

Algorithm 1 provides the process to obtain thedistribution of the intermediate tuples for each reducetask. For a specific MapReduce job, this algorithmfirst starts this job with the sample data and recordsthe number of tuples from a map local node to eachreduce task based on a monitor in each map node. Asa practice, a data cluster is the subset of all pairs withthe same key in this paper, and all clusters processedby the same reducer constitute a partition [6]. Thefunction getOrignalMapNode is used to retrace theintermediate tuples and obtain the map nodes thatproduce these data [30]. From Algorithm 1, becausethe intermediate tuple distribution of sample dataremains coherent with the whole input dataset, wecan calculate the data size of reduce tasks from everymap node under the consistent distribution law.

Obviously, there is a trade-off between the sam-pling overhead and the accuracy of the result. Theexperiments in Section 6.2.1 are designed to select anappropriate sample rate that satisfies these seeminglycontradictory necessities. However, it is important tonote that not all jobs must sample the data before



6

running. The key distribution is the objective, whichis indeed application independent. For jobs with thesame input, after uploading the original data to theHadoop Distributed File System (HDFS), their origi-nal storage distributions in data nodes are relativelyfixed.

Algorithm 1 Distribution DetectionInput:

The sample of data blocks BS;a MapReduce job: mrj ;the number of computing nodes N ;the number of reduce tasks R;

Output:The matrix C of whole input data.run the MapReduce job mrj using BS as the input data;initialize an intermediate matrix C .Cσ,l

i,j ←0, 0≤ l < N , 0≤ j < R;for each cluster ck with tuple key k do

//get reducer serial number for ck ;j ← systemhash(k);for each map node l do

for each map task i do//if ck is come from node lif l=getOrignalMapNode(ck) then

Cσ,li,j ← Cσ,l

i,j + tuple number of(ck);end if

end forend for

end forSN ← the number of tuples in BS;WN ← the number of tuples in whole input data;for each map node l do

for each map task i dofor each reduce task j do

Cσ,li,j ← (Cσ,l

i,j ×WN ) / SN ;end for

end forend forreturn matrix C =

{Cσ,l

i,j

}.

For the matrix C = Cσ,li,j , the number and position

of map tasks on the nodes are simply up to thesizes and distributions of the input data, which candetermine subscripts i and l for the element in matrixC. Subscript j is just the order number of the reducetasks, but the number of reduce task can actually bepre-set in the source code. Hence, for this situation,we must perform data sampling only once, and theobtained matrix C can be reused by different jobs.

5 COMMUNICATION ORIENTED REDUCEPLACEMENT5.1 The Model of MapReduce

In most implementations of various Hadoop ver-sions, the key-value pairs space is partitioned amongthe reducers. The partitioner design has a direct im-pact on the overall performance of the job: a poorlydesigned partitioning function will not evenly dis-tribute the load over the reducers. As the defaultpartitioner, HashPartitioner hashes a record key todetermine the partition (and thus which reducer) inwhich the record belongs. The number of partitions isthen equal to the number of reduce tasks for the job[31].

Let n be the number of nodes, and m be thenumber of map tasks. We first initialize the node set

as {N0, N1, · · ·, Nl, · · ·, Nn−1}, and the map set as{M0,M1, · · ·,Mi, · · ·,Mm−1}, 0 ≤ n ≤ m for rack setRr, r ∈ {0, 1, ···, k−1}, 0 ≤ k < n. For this model, somespecific data structures can be formalized as follows:

(1) V : A vector of length p whose elements indicatethe relevant number of key/value tuples in everynode. If vl,j = k, there are k key/value pairs in nodeNl assigned to reduce j. Therefore,we have

vl,j =

m−1∑i=0

Cli,j , 0 ≤ l < n (7)

and

Vl = [vl,0, vl,1, · · ·, vl,p−1], C = [V0, V1, · · ·, Vn−1] (8)

(2) D: A matrix of n × n that defines the distancebetween physical nodes. According to the networklatency, we can define the distance between two phys-ical nodes in the same rack as d1, the distance betweentwo physical nodes in different racks but in the samecluster as d2, the distance between two physical nodesin different clusters as d3, and the distance betweentwo physical nodes in different data centers as d4. Thedistance among maps is the distance between the twonodes in which the maps are located; i.e., the distanceamong maps in the same node is 0. In the four situ-ations above, the distance value would increase withincreasing physical distance: 0 < d1 < d2 < d3 < d4.This paper supposes that the shorter the distance, thefaster the data transfer speed, which is the theoreticalbasis of the model optimization.

(3) R: A matrix of p × n that defines the positionof reduce tasks started on the node, and is a typicalsparse matrix. The element rl,j is a Boolean variablethat indicates whether reduce task j is set up on nodel. The following condition ensures that each reducetask should be placed on only one node:∑

l

rl,j = 1, ∀l ∈ [0, n− 1] (9)

The quantity of reduce tasks is usually less thanthat of map tasks. A reduce task can be started in aphysical node only if the physical node can providesufficient computing resources apart from the existingmap tasks. In this model, the allocation matrix Rindicates the position at which to start up the reducetasks, meanwhile, another matrix D defines the mutu-al distances between the physical nodes. On this basis,this model uses a communication matrix T amongnodes to quantify the cost of data transmission calledintermediate results which are copied from map tasksto reduce tasks in cluster. On this basis, we use avector RVj to denote the position of a task on a nodein the matrix R:

RVj = [r0,j , r1,j , · · ·, rl,j , · · ·, rn−1,j ]. (10)

The element rl,j in vector RVj is a Boolean variablethat indicates whether reduce task j is placed on node



7

l. Therefore, if there are r reduce tasks among nodes,we can define matrix R as follows:

R = [RV0, RV1, · · ·, RVr−1]T (11)

To capture locality, we define a communication costfunction that measures the data transfer between twonodes. In most Hadoop implementations, the largestcost of a MapReduce application is in copying innerresult data from map tasks to relevant reduce tasksthat are placed on the other physical nodes. Usingthe network hops as the near-far measure, this paperdefines a vector DV to calculate the distance amongnode Nl and other nodes:

DVl = [disl,0, disl,2, · · ·, disl,d, · · ·, disl,n−1]0 < |disl,d| ≤ d4, 0 ≤ l, d < n.

(12)

For the process of map tasks outputting the in-termediate key-value set in their local node and therelated reduce tasks of fetching the correspondingdata through the network, there are two key factorsregarding cost matrix T : the amount of data trans-ferred and the distances between the map nodes andreduce nodes. An element tl,j of the cost matrix canbe obtained as follows:

tl,j = DVl ×RV Tj , ∀l ∈ [0, n− 1], ∀j ∈ [0, p− 1]

=n−1∑d=0

disl,d × rd,j(13)

In the foregoing discussion, Cl is defined as an in-termediate result allocation matrix, which representsthe key/value pairs distribution in node l. tkl,j denotesthe kth placement choice for reduce task j on node l.With cost matrix T , the minimal cost MC of the wholeHadoop system can be calculated by multiplying thedistribution matrix C by the cost matrix T from adistinguished central node:

MC(Cσ,l, T

)= min

k

m−1∑i=0

p−1∑j=0

cσ,li,j

× tkl,j

(14)

From Eq. (14), in the data skew environment, ifmost data for reduce task j come from node l, whenMC obtains the minimal value, the jth reduce task canfetch the largest local data blocks in the kth computingnode. Here, m is the number of the map tasks, andp is the number of partitions. To obtain the loadbalance of all reduce tasks, Algorithm 2 is designed tocombine the smaller clusters of < key, value > tuplesto an optimal reduce task, according to the currentworkload of the reduce tasks. Algorithm 2 adjuststhe intermediate data distribution matrix C which isobtained from Algorithm 1 before the system runs.The measure of the FoS value for typical benchmarks

in the experiments also verifies that this algorithm canmaintain the load balance for M/R tasks effectively.

Algorithm 2 Cluster CombinationInput:

A collection of tuple set: TS = {ts1, · · ·, tsk, · · ·, tsK};the collection of reduce tasks: R={r1, r2, · · ·, rj , · · ·, rp}, p ≤ K;the predicted matrix C of whole input data from algorithm 1.

Output:The adjusted matrix C of input data.calculate the size |tsi| of each cluster. 1≤ i ≤ K;sort TS in descending order according to | tsk |;assign p selected clusters from TS sequentially to p reducers;for each k ∈ [p + 1,K] do

for each map task i doassign cluster tsk to reducer rp;//get the map node for a specific reduce task;l ← getOrignalMapNode(tsk);j ←p;//adjust matrix C as output;Cl

i,j ← tuple number of(tsk);sort R in descending order;

end forend forreturn matrix C.

To achieve an optimal reduce task placement solu-tion that can minimize the network communicationoverhead among different physical nodes, the objec-tive function of this model can be specified as thefollowing optimization problem:

Minimize∑m,n

MC(Cσ,l, T ) (15)

where n denotes the number of nodes. Hence, thereduce task placement can be specialized as a prob-lem to obtain the assignment of matrix T , whichcan achieve the target of Eq. (15). The Cost MatrixGeneration algorithm shows the specific steps to cal-culate cost matrix T by multiplying distribution C bydistance matrix D.

5.2 Placement Algorithm for Reduce TasksIn Algorithm 3, the getPartiionsList() method re-

turns list partitions on the lth node.

Algorithm 3 Cost Matrix GenerationInput:n: the number of node;C: split-partition matrix;D: distance of resources.

Output:The cost matrix T .partitionList← ∅;//traverse each element in matrix C;for each l ∈ [0, n− 1] do

PartitionList=getPartitionsList(C, i);for j in partitionList do

//Count pairs in every partition at node l;tempC[j]← Calculate(C, j,D);

end fortempT ← tempC × getDist(D, j);T ← Com(tempQ);continue;

end forreturn T .

Parameter D is the distance matrix. The methodCalculate() returns the value with the number of



8

intermediates in node nl, and Com() returns theminimum value of the array.

In Algorithm 4, the getNodesList() method returnsthe list of node, and getCost() will obtain the costof placing the jth reduce task on the lth node. Theminimum value can be selected in the array by themethod Minimize().

Algorithm 4 Reduce Task PlacementInput:p: the number of reduce;n: the number of node;T : the cost matrix.

Output:The placement queue R which able to minimize the value of MC.List nodesList← ∅;for each j ∈ [0, p− 1] do

nodesList← getNodesList(N, T );for l in nodes list do

//obtain the cost on lth node;tempSum[l]← getCost(T, j, l);

end forR[l]←Minimize(tempSum);

end forreturn R.

6 EXPERIMENTAL EVALUATION

The experiments in this paper are divided in-to two steps. First, we provide a detailed micro-benchmarking for our data and put forward reduceplacement techniques for each MapReduce job classon a real cluster composed of 20 physical machines.Second we present a detailed and quantitative analy-sis of the result for mixed job types executed underthis reduce placement algorithm.

6.1 Experiment Setting

In our experiments, the following reduce schedul-ing algorithms are chosen for comparison.

NorP (Normal Placement). In original Hadoop im-plementations, reduce tasks are launched according torandom assignment and the resource utilization in thecomputing nodes. In Hadoop version 2.0 or higher,this distribution can be controlled by the programmerin the YARN framework, which is the implementationmechanisms of CORP: to complete the reduce place-ment by invoking these YARN APIs [4].

Range (Range Partition). This is a widely usedalgorithm of partition distribution. In this method, theintermediate < key, value > tuples are first sorted bykey, and then the tuples are sequentially assigned tothe reduce task according to this key range. Becausethe partitions may be split or combined in to differentreduce tasks, this algorithm can ensure the furthestload balance of the reduce tasks [6], [25].

SARS (Self-Adaptive Reduce Scheduling). This isan optimal reduce scheduling policy for reduce tasksstart time in the Hadoop platform [19]. It can decidethe start time points of each reduce task dynamical-ly according to each job context, including the task

completion time and the size of the map output.This model can decrease the reduce completion timeand the system average response time in the Hadoopplatform effectively [19].

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

Fig. 4: Experiment network topology

As shown in Fig. 4, our cluster consists of 20physical machines loaded with the operating systemof Ubuntu 12.04 (KVM as the hypervisor) with 16 core2.53 GHz Intel processors. Those machines are man-aged by CloudStack, which is an open source cloudplatform with two racks, each of which contains 10physical machines. The network bandwidth is 1 Gbps,and the nodes within a rack are connected througha single switch. In these experiments, the volume ofintermediate data transmissions of the whole systemon this core switch can be counted by the monitor ofthe network management software. Each job uses acluster of 50 VMs with each VM configured with 4 GBof memory and four 2 GHz vCPUs. A description ofthe various job types and the dataset sizes are shownin Table 2.

TABLE 2: Job types and the dataset sizes

Workload classification Benchmarks Input data

Map and Reduce-input heavy sort 10G

Map-input heavy Grep:word search 10G

Reduce-input heavy Join 2G

6.2 Performance Evaluation

6.2.1 Sampling Experiments

In this section, we first propose an evaluation for-mula as Eq. (16) to select the appropriate sample rate,which can comprehensively consider the importanceof cost, effect, and variance in sampling.

i = argMin[fi(∆i, Ti,Φi) = α∆i + βTi + γΦi] (16)

where function fi(∆i, Ti,Φi) is a comprehensive indexconsidering both cost and effect, in which ∆i reflectsthe difference among the sequences of FoS valuesbetween the currently adopted percentage and 100%(the whole input dataset), which can be calculated byEq. (17):



9

∆i =

√√√√√ N∑j=1

di,j −1

N

N∑j′=1

d5,j′

2

(17)

where N denotes the experimental repetition time,and di,j represents the FoS value obtained in thejth sampling experiment under the ith sampling rate.1 ≤ i ≤ SN is the order number of different samplingpercentages: {1%, 25%, 50%, 75%, 100%}, and SNdenotes the space size of different sampling rates. Forthis experiment, SN = 5, and d5,j denotes the valueswith a 100% sampling rate.

As an average sampling execution time, Ti can becalculated simply by Eq. (18):

Ti =1

N

N∑j=1

ti,j (18)

where ti,j represents the execution time of the jthsampling experiment under the ith sampling rate,1 ≤ i ≤ SN and 1 ≤ j ≤ N .

To full consider the influence of data volatilities, Eq.(19) provides the process to calculate the parameter Φi

based on the standard deviation formula:

Φi =

√√√√√ 1

N

N∑j=1

di,j −1

N

N∑j=1

di,j

2

(19)

Fig. 5 shows the FoS and execution time obtainedin times sampling experiments. Based on these results,table 3 provides the final values of ∆i, Ti, and Φi

for all benchmarks with various sampling rates. Eachgroup of sampling experiments is repeated ten times,which means that the parameter N in Eq. (17), Eq. (18)and Eq. (19) should be set to 10 in these experiments.Finally, for the weight coefficients which reflect theimportance of cost, effect, and variance in sampling,we can simply set α = β = γ = 1. That is, we thinkthat the cost, effect, and data volatility are equallyimportant.

1 2 3 4 5 6 7 8 9 10

40

80

120

160

200

240

280

FoS

(%)

Sampling Frequency

1% 25% 50% 75% 100%

200

300

400

500

600

700

800

900

1000

Exe

cutio

n Ti

me(

0.1

sec)

Fig. 5: The comparison experiment with varioussampling rate.

TABLE 3: Comprehensive evaluation for cost andeffect with different benchmarks

i rate ∆i Ti Φi fi

1 1% 407.307 316 51.173 774.48

2 25% 209.564 434 38.833 682.397

3 50% 143.043 555 31.668 729.711

4 75% 80.188 663 14.925 758.113

5 100% 11.580 730 3.662 745.242

Moreover, what is noteworthy is the group of exper-iments with a sample rate of 1%: in this case, the timecosts are relatively low, and most of the FoS valuesare lower than the other sample rates. However, itis easy to find that the experimental results of FoSvalues and time costs are very volatile, and the resultsof ∆i in Table 3 confirm that there is great differencecompared with FoS values with a 100% sample rate,which are actually measured by Eq. (17). We holdthat the lower sample rate cannot easily represent theaccurate distribution of the whole input dataset.

By comprehensive consideration according to Eq.(16), it is easy to learn that sampling 25% of the maptasks is an appropriate choice for the input data ofSort and Grep benchmarks. For benchmark join, themost appropriate sample rate is 1%.

As mentioned previously, the major motivation forCORP is to improve the data locality to diminishcross-rack communication. In the following experi-ments, to verify the advantages of CORP, we evaluatethis model for FoS and job execution time using thesecommon benchmarks: Sort, Grep, and Join.

As mentioned in Section 4.2.1, for the jobs with thesame input, we need only to perform data samplingonce, and the obtained matrix C can be reused bydifferent jobs. Moreover, because the sampling targetis merely detects the distribution of intermediate data,the time cost is much lower than the practical jobs.

6.2.2 Sort Benchmark TestingThe Sort benchmark in Hadoop is usually used for

workload testing because it is a heavy job for map andreduce inputs. In these experiments, we generate 10GB of synthetic data sets following Zipf distributionswith σ parameters varying from 0.1 to 1.2 to controlthe degree of skew. The reason to choose Zipf todescribe the frequency of the intermediate keys is thatthis distribution is very common in data coming fromhuman society [26].

In the following experiments, the performance e-valuations of the relevant algorithms are illustratedconsidering their job execution time in the change ofthe input data skew degree. In this paper, the loadbalancing and skew degree are measured by the factorof skew. Fig. 6 shows the experimental results basedon the 10 GB of synthetic data. From Fig. 6(a), we canconclude that CORP can improve the FoS obviouslyfor different input data with various skew degrees.



10

FosS

[%]

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.20

20

40

60

80

100

120

140

160

180

200

Skew Degree( )

NorP SARS CORP Range

(a) load balance

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1400

500

600

700

800

900

1000

1100

1200

Job

Exc

ute

Tim

e(se

c)

Skew Degree( )


(b) job execution time

0.2 0.4 0.6 0.8 1.0 1.230

40

50

60

70

80

90

100

Cro

ss-tr

anfe

r[% in

put s

ize]

Skew Degree( )


(c) inner data traffic

Fig. 6: Performance vs. data skew for Sort.

The curves in Fig. 6(b) show that the job executionperformance of CORP remains better than normalplacement but worse than SARS. Through this result,we can know that CORP can decrease the executiontime in the reduce phase because it can make theintermediate results more localized.

More specifically, as we can see in Fig. 6(a), thevalue of FoS increases rapidly when the degree ofskew exceeds 0.7 for all the experimental algorithms,and from Fig. 6(b), the execution times of all algo-rithms also increase substantially once the degree ofskew reaches a certain threshold. Through the reduceplacement, CORP ensures even data processing in thenodes. From the results, this algorithm has better per-formance in terms of FoS optimization, but it performspoorly in terms of execution time.

The results in Fig. 6 shows that, by sampling todetermine the placement of reduce tasks, CORP canobtain a better data load balance, lower FoS and lesstransfer in the cross-rack compared with other reducescheduling algorithms. However, when the skew de-gree is less than 0.7, CORP performs worse than NorPand SARS in terms of execution time. The reason isthat CORP must calculate the cost matrix and makethe decision on reduce task placement after running aseparate sampling job, which would incur significateextra overheads. However, because the data trans-mission among different nodes is optimized throughreduce local placement in CORP, the whole executiontime of a job can be decreased to offset the time spent,which is increased by the extra overhead for decision-making. For this reason, although the performanceof CORP and Range are both lower among thesealgorithms, CORP has a much lower execution timethan Range.

A similar trend is seen in Fig. 6(c), in which the barsillustrate the transfer data through the core switch asa percent of overall inner data traffic. As the skewdegree increases, an approximate task location pro-duced by CORP can help balance the expected loadamong physical machines and increase data localitycompared with other algorithms, which focus onlyon the load balancing without consideration givento data locality. The data transmission of CORP on

the core switch have been decreased by up to 51.9%compared with NorP with the skew degree set toσ = 1.1 (in the Fig. 6(c)).

Fig. 7 represents the variation of job execution timewhen the data size ranges among {4 GB, 8 GB, 12GB} with different skew degrees (σ=0.1 and σ=1.2).When the data set has a smaller skew degree (seeFig. 7(a)), SARS is the most time-efficient algorithm,and CORP performs better than only NorP owing tothe overhead for data sampling and task placementdecision-making.

4 8 12

200

400

600

800

1000 NorP SARS CORP Range

Job

Exc

ute

Tim

e(se

c)

(a) Data Size GB =0.14 8 12

0

400

800

1200

1600


Job

Exc

ute

Tim

e(se

c)

(b) Data Size GB =1.2

Fig. 7: Performance vs. data size for sort

With the increased skew degree and the data size(see Fig. 7(b)), the growth rate of the CORP executiontime is smaller than the others. However, in thesetwo experiments, Range has the worst execution timeowing to its even splitting and combining strategy forintermediate tuples. This causes poor locality in thereduce phase and the redundant inner communicationfinally results in the longer execution times. Especiallywhen the data-set size gradually approaches 12 GB,the execution time of CORP becomes smaller thanSARS with skew degree σ = 1.2. The reason is that theappropriate location of reduce tasks can help achievemuch greater localization, and decrease the unneces-sary communication among racks. This experimentalso demonstrates that the performance of CORP isrelatively high when the scales of the skew degreeand data set are large.

Fig. 8 illustrates the relationship between FoS andthe data size: the smaller the FoS, the better the loadbalancing (keeping σ = 1.2). When the data set hasa smaller skew degree and data size varies from 4GB to 12 GB (see Fig. 8(a)), we can observe that theFoS values of all methods increase slowly, but thisindicator of CORP is always better than the othercompared algorithms. As the dataset scale increases



11

rapidly, the performance of CORP is more prominent:the FoS in our system is 60% smaller than NorP in theHadoop implementation.

As shown in Fig. 8(b), the factor of skew amongall reduce tasks in CORP is only 160%, whereas, itreaches 280%, 260%, and 230% in NorP, SARS, andRange, respectively.

4 8 120

5

10

15

20

25

30

35


FoS

[% ]

(a) Data Size GB =0.14 8 12

120

160

200

240

280


FoS

[% ]

(b) Data Size GB =1.2

Fig. 8: FoS vs. data size for Sort.

6.2.3 Grep Benchmark Testing

Grep is a popular application for large-scale dataprocessing with heavy map-input. It searches someregular expressions through input text files and out-puts the lines that contain the matched expressions.We improve the Grep benchmark in Hadoop so that itoutputs the matched lines in a descending order basedon how frequently the searched expression occurs.The data set we used is the full English Wikipediaarchive with a total data size of 10 GB, which consti-tute the data processing jobs with heavy map tasks.

Because the behaviour of Grep depends on howfrequently the search expression appears in the inputfiles, we tune the expression and make the inputquery percentages vary from 10% to 100%. Fig. 9shows the changes of job execution time, FoS valueand cross-rack transfer with increasing query percent-age. Note that most current versions of Hadoop do notprovide a suitable range partition for this application:their pre-run sampler can detect the input data butcannot handle applications in which the intermediatedata are in a different format from the input.

In contrast, the proposed sample algorithm in thispaper can spot-check the intermediate data directlyand works well for all types of applications. As we cansee in Fig. 9(a), CORP performs obviously better thanNorP, SARS, and Range with lower query percentage.This is because CORP has obvious advantages forsearching unpopular words in the archive and tendsto generate results with heavy data skew. Althoughthe curve of execution time is always lower than thoseof other algorithms in this experiment, as the querypercentage increases, because the distribution of theresult data becomes increasingly uniform, the perfor-mance gap rapidly closes. As a matter of fact, whenthe query percentage approaches 100%, as shown inthe Fig. 9(b), the performance of CORP is very similarto the other three algorithms.

The inner data transfer in the Fig. 9(c) shows thatCORP has lower traffic (the highest is only 30.1%)

compared with the NorP, SARS, and Range algorithmsfor various query percentages.

6.2.4 Join Benchmark TestingJoin is one of the most common reduce-input heavy

applications, especially in a data skew environment.We implement a simple broadcast Join job in Hadoopthat partitions a large table in the map phase, whereasa small table is directly read in the reduce phase togenerate a hash table to speed up the Join operation.When the small table is too large to fit into thememory, we use a buffer to maintain only a part of thesmall table in memory and use the cache replacementstrategy to update the buffer. For this experiment, weselect a data set with size = 2 GB and σ = 1.2 fromthe widely used corpora ”Yahoo News Feed dataset,version 1.0” [32] to evaluate the time performance andload balance effect. CORP is compared with otheralgorithms under Hash Join (PHJ) and Replicated Join(PRJ) in Pig [33]. Fig. 10(a) shows the load balanceand job execution time of these three test cases. InFig. 10(a), the best Join scheme in Pig is PRJ, whichsplits the large table into multiple map tasks andperforms the Join in map tasks by reading the smalltable directly into memory.

In Fig. 10(b), the best scheme in Pig is PHJ, whichsamples a large table to generate the key distributionand makes the partition decision beforehand. In Fig.10(c), the difference in job execution time can beexplained by inner data network traffic. In CORP, thedata can be read from closer nodes, and lower timecost of communication can decrease the whole jobexecution time. However, because the data samplingwill be additional works that can incur extra run time,the job execution speed in CORP is certainly lowerthan that of SARS.

6.2.5 Experiment SummaryIn the foregoing experiments, we evaluate the per-

formance and effect using input data with even vari-able skew degrees. To verify the effectiveness of syn-thesized data for Sort and Join, a group of comparativeexperiments are designed with the results shown inFig. 11. For the chosen realistic data, we generate somedifferent sizes of synthesized data with similar skewdegree and repetition of keys. In this example, σ = 0.7.From Fig. 11, we can easily see that the experimentalresults of FoS and execution time on synthesized dataare roughly identical to the real loads.

The squares in Fig. 12 summarize the comparisonsamong these algorithms: NorP, SARS, CORP, andRange. The results in Fig. 12(a) show that CORP canachieve the shortest execution time for the smallestdata communication from map tasks to reduce tasks.The comparison values of FoS in Fig. 12(b) furtherillustrate that, benefiting from the data-locality, CORPcan achieve better load balance than other algorithmswithin the workloads: Grep, Sort, and Join.



12

10 20 30 40 50 60 70 80 90 1000

30

60

90

120

150

180

210

240

270

300

330

360

390Fo

s[%

]

Query Percentage[%]

CORP NorP SARS Range

(a) load balance

10 20 30 40 50 60 70 80 90 100

200

300

400

500

600

700

800

Exe

cutio

n Ti

me(

sec)

Query Percentage[ % ]

CORP NorP SARS Range


0.2 0.4 0.6 0.8 1.0

20

40

60

80

Cro

ss-tr

anfe

r[% in

put s

ize]

Query Percentage



Fig. 9: Performance vs. query percentage for Grep.

60

120

180

240

300

360

Replicated JoinHash Join

FoS[%]


(a) load balance

0

300

600

900

1200

1500

1800


Exec

utio

n Ti

me(

sec)



0

30

60

90

120

150

180


Cro

ss-tr

anfe

r[% in

put s

ize]



Fig. 10: Performance vs. Hash and Replicate for Join.

1GB 2GB

100

200

300

400

500

600

700

HashJoin-1(real) HashJoin-2(syn) Sort-1(real) Sort-2(syn)

HashJoin-1(real) HashJoin-2(syn) Sort-1(real) Sort-2(syn)

Exe

cutio

n Ti

me(

sec)

(a) =0.7 1GB 2GB

20406080100120140160180200

Fos(

%)

(b) =0.7

Fig. 11: Effect evaluation for real and synthesis data.

Sort Grep Hash Join

300

600

900

1200


Exe

cutio

n Ti

me(

sec)

(a) ( =1,size=2G,QP=50%)Sort Grep Hash Join

40

80

120


Fos(

%)

(b) ( =1,size=2G,QP=50%)

Fig. 12: Whole performance evaluation.

The final group of experiments is to test the per-formance of batch jobs. Because one job client alwayswaits for the execution to complete after the job issubmitted, our way is to submit various numbers ofjobs using multiple shell clients at the same time. Theexecution time is the duration from the start of thefirst job to the end of the last job. Fig. 13 records theexecution time under CORP for batch jobs using thebenchmarks: Sort, Join, and Grep. It is easy to see thetime cost always sharply increases with more than 6concurrent jobs.

1 2 3 4 5 6 7 8 9 10

0

500

1000

1500

2000

2500

3000

3500

4000

Sort Grep

Exec

utio

n Ti

me(

sec)

(a) ( =1,size=2G,QP=50%)1 2 3 4 5 6 7 8 9 10

020004000600080001000012000140001600018000

Join

Exec

utio

n Ti

me(

sec)

(b) Hash Join ( =1,size=2G)

Fig. 13: The performance evaluation for batch jobs.

7 CONCLUSION

The existence of data skew in the intermediateoutput created by map tasks provides an opportunityto optimize the cross-rack communication by placingthe reduce tasks on the appropriate nodes. This papermainly involves the following work, i.e., a samplingmethod based on a reservoir algorithm is applied tosample the data selection. We propose an evaluationmodel and undertake a great deal of experimentalresearch to select the appropriate sample rate. Theadvantage of this model is the ability to comprehen-sively consider the importance of cost, effect, andvariance in sampling.

Sampling is an independent MapReduce job, whichcan output a distribution matrix of intermediate re-sults in each partition. Based on this, by calculatingthe distance and cost matrices among the cross-nodecommunication, the related map and reduce tasks canbe scheduled to relatively nearby physical nodes fordata locality. Experiments verify that the inner datatransmission can be obviously optimized through thisalgorithm. For most highly skewed data, the job exe-



13

cution time can also be decreased for lower inner datacommunication.

ACKNOWLEDGMENTS

The work is supported by the National Nat-ural Science Foundation of China (Grant Nos.61572176), National High-tech R&D Program of Chi-na (2015AA015303), the Key Technology Researchand Development Programs of Guangdong Province(2015B010108006), and International S&T CooperationProgram of China (2015DFA11240).

REFERENCES

[1] N. Tiwari, S. Sarkar, and U. Bellur, “Classificationframework of mapreduce scheduling algorithm-s,” Acm Computing Surveys, vol. 47, no. 3, pp. 1–38, 2015.

[2] X. Lu, F. Liang, B. Wang, L. Zha, and Z. X-u, “Datampi: Extending mpi to hadoop-like bigdata computing,” IEEE International Parallel &Distributed Processing Symposium, 2014.

[3] L. Jiang, P. Patel, G. Ostrouchov, and F. Jamitzky,“Openmp-style parallelism in data-centered mul-ticore computing with r,” Acm Sigplan Notices,vol. 47, no. 8, pp. 335–336, 2014.

[4] “Hadoop.[eb/ol].” http://hadoop.apache.org.[5] Q. Chen, C. Liu, and Z. Xiao, “Improving mapre-

duce performance using smart speculative exe-cution strategy,” Computers, IEEE Transactions on,vol. 63, no. 4, pp. 954–967, 2014.

[6] Q. Chen, J. Yao, and Z. Xiao, “Libra: Lightweightdata skew mitigation in mapreduce,” Parallel andDistributed Systems, IEEE Transactions on, vol. 26,no. 9, pp. 2520–2533, 2015.

[7] J. Tan, S. Meng, X. Meng, and L. Zhang, “Im-proving reducetask data locality for sequentialmapreduce jobs,” in INFOCOM, 2013 ProceedingsIEEE. IEEE, 2013, pp. 1627–1635.

[8] M. Bourguiba, K. Haddadou, I. El Korbi, andG. Pujolle, “Improving network i/o virtualiza-tion for cloud computing,” Parallel and DistributedSystems, IEEE Transactions on, vol. 25, no. 3, pp.673–681, 2014.

[9] L.-Y. Ho, J.-J. Wu, and P. Liu, “Optimal algo-rithms for cross-rack communication optimiza-tion in mapreduce framework,” in Cloud Comput-ing (CLOUD), 2011 IEEE International Conferenceon. IEEE, 2011, pp. 420–427.

[10] M. Isard, V. Prabhakaran, J. Currey, U. Wieder,K. Talwar, and A. Goldberg, “Quincy: fairscheduling for distributed computing clusters,”in Proceedings of the ACM SIGOPS 22nd symposiumon Operating systems principles. ACM, 2009, pp.261–276.

[11] F. Ahmad, S. Lee, M. Thottethodi, and T. Vijayku-mar, “Mapreduce with communication overlap

(marco),” Journal of Parallel and Distributed Com-puting, vol. 73, no. 5, pp. 608–620, 2013.

[12] N. Maheshwari, R. Nanduri, and V. Varma,“Dynamic energy efficient data placement andcluster reconfiguration algorithm for mapreduceframework,” Future Generation Computer Systems,vol. 28, no. 1, pp. 119–127, 2012.

[13] S. Ibrahim, H. Jin, L. Lu, S. Wu, B. He, and L. Qi,“Leen: Locality/fairness-aware key partitioningfor mapreduce in the cloud,” in Cloud ComputingTechnology and Science (CloudCom), 2010 IEEE Sec-ond International Conference on. IEEE, 2010, pp.17–24.

[14] M. Hammoud and M. F. Sakr, “Locality-awarereduce task scheduling for mapreduce,” in CloudComputing Technology and Science (CloudCom),2011 IEEE Third International Conference on. IEEE,2011, pp. 570–576.

[15] J. Tan, S. Meng, X. Meng, and L. Zhang, “Im-proving reducetask data locality for sequentialmapreduce jobs,” in INFOCOM, 2013 ProceedingsIEEE. IEEE, 2013, pp. 1627–1635.

[16] B. Palanisamy, A. Singh, L. Liu, and B. Jain,“Purlieus: locality-aware resource allocation formapreduce in a cloud,” in Proceedings of 2011International Conference for High Performance Com-puting, Networking, Storage and Analysis. ACM,2011, p. 58.

[17] Y. Kwon, M. Balazinska, B. Howe, and J. Rolia,“Skewtune: mitigating skew in mapreduce appli-cations,” in Proceedings of the 2012 ACM SIGMODInternational Conference on Management of Data.ACM, 2012, pp. 25–36.

[18] K. Wang, X. Zhou, T. Li, D. Zhao, M. Lang, andI. Raicu, “Optimizing load balancing and data-locality with data-aware scheduling,” in Big Data(Big Data), 2014 IEEE International Conference on.IEEE, 2014, pp. 119–128.

[19] Z. Tang, L. Jiang, J. Zhou, K. Li, and K. Li,“A self-adaptive scheduling algorithm for reducestart time,” Future Generation Computer Systems,vol. 43, pp. 51–60, 2015.

[20] R. P. Mount et al., “The office of science data-management challenge,” Stanford Linear Accel-erator Center (SLAC), Tech. Rep., 2005.

[21] Y. Xu and P. Kostamaa, “Efficient outer join dataskew handling in parallel dbms,” Proceedings ofthe VLDB Endowment, vol. 2, no. 2, pp. 1390–1396,2009.

[22] E. Ardizzoni, A. A. Bertossi, M. C. Pinotti,S. Ramaprasad, R. Rizzi, and M. V. Shashanka,“Optimal skewed data allocation on multiplechannels with flat broadcast per channel,” Com-puters, IEEE Transactions on, vol. 54, no. 5, pp.558–572, 2005.

[23] J. W. Stamos and H. C. Young, “A symmetricfragment and replicate algorithm for distribut-ed joins,” Parallel and Distributed Systems, IEEE



14

Transactions on, vol. 4, no. 12, pp. 1345–1354, 1993.[24] S. Acharya, P. B. Gibbons, and V. Poosala, “Con-

gressional samples for approximate answeringof group-by queries,” in ACM SIGMOD Record,vol. 29, no. 2. ACM, 2000, pp. 487–498.

[25] J. W. Kim, S.-H. Cho, and I. Kim, “Improvingefficiency in range-based query processing on thehadoop distributed file system by leveraging par-titioned tables,” International Information Institute(Tokyo). Information, vol. 17, no. 10 (B), p. 5311,2014.

[26] B. Gufler, N. Augsten, A. Reiser, and A. Kemper,“Load balancing in mapreduce based on scal-able cardinality estimates,” in Data Engineering(ICDE), 2012 IEEE 28th International Conference on.IEEE, 2012, pp. 522–533.

[27] J. Tan, X. Meng, and L. Zhang, “Coupling taskprogress for mapreduce resource-aware schedul-ing,” in INFOCOM, 2013 Proceedings IEEE. IEEE,2013, pp. 1618–1626.

[28] J. Lin et al., “The curse of zipf and limits toparallelization: A look at the stragglers problemin mapreduce,” in 7th Workshop on Large-ScaleDistributed Systems for Information Retrieval, vol. 1,2009.

[29] “Reservoir sampling.[eb/ol].” in http-s://en.wikipedia.org/wiki/Reservoir sampling.

[30] R. Grover and M. J. Carey, “Extending map-reduce for efficient predicate-based sampling,” inData Engineering (ICDE), 2012 IEEE 28th Interna-tional Conference on. IEEE, 2012, pp. 486–497.

[31] F. Yuanquan, W. Weiguo, X. Yunlong, andC. Heng, “Improving mapreduce performance bybalancing skewed loads,” Communications, China,vol. 11, no. 8, pp. 85–108, 2014.

[32] “Ratings and classification data. [eb/ol].” inhttp://webscope.sandbox.yahoo.com.

[33] C. Olston, B. Reed, U. Srivastava, R. Kumar, andA. Tomkins, “Pig latin: a not-so-foreign languagefor data processing,” in Proceedings of the 2008ACM SIGMOD international conference on Manage-ment of data. ACM, 2008, pp. 1099–1110.

Zhuo TANG received the Ph.D. in computerscience from Huazhong University ofScience and Technology, China, in 2008.He is currently an associate professorof the College of Computer Science andElectronic Engineering at Hunan University,and is the associate chair of the departmentof computing science. His majors aredistributed computing system, cloudcomputing, and parallel processing for bigdata, including distributed machine learning,

security model, parallel algorithms, and resources scheduling andmanagement in these areas. He is a member of ACM and CCF.

Wen Ma received his B.A.Sc. degreein computer science from the Universityof Science and Technology Liaoning,China. And now he is working towardsthe master degree at the College ofInformation Science and Engineering,Hunan University, China. His researchinterests include parallel computing,improvement and optimization of taskscheduling module in Hadoop and Sparkplatforms.

Kenli Li received the Ph.D. in computerscience from Huazhong University ofScience and Technology, China, in 2003.Now he is a professor of ComputerScience and Technology at HunanUniversity, associate director of NationalSupercomputing Center in Changsha. Hismajor research includes parallel computing,grid and cloud computing, and DNAcomputer. He has published over 320journal articles, book chapters, and refereed

conference papers. And he is an outstanding member of CCF and amenber of IEEE.

Keqin LI is a SUNY Distinguished Professorof computer science. His current research in-terests include parallel computing and high-performance computing, distributed comput-ing, energy-efficient computing and commu-nication, heterogeneous computing systems,cloud computing, big data computing, CPU-GPU hybrid and cooperative computing, mul-ticore computing, storage and file system-s, wireless communication networks, sensornetworks, peer-to-peer file sharing systems,

mobile computing, service computing, Internet of things and cyber-physical systems. He has published over 410 journal articles, bookchapters, and refereed conference papers, and has received severalbest paper awards. He is currently or has served on the editorialboards of IEEE Transactions on Parallel and Distributed Systems,IEEE Transactions on Computers, IEEE Transactions on Cloud Com-puting, Journal of Parallel and Distributed Computing. He is an IEEEFellow.

Date post:	20-May-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

A Data Skew Oriented Reduce Placement Algorithm Based on...

Documents