1

Cost Minimization for Big Data Processing inGeo-Distributed Data Centers

Lin Gu, Student Member, IEEE , Deze Zeng, Member, IEEE , Peng Li, Member, IEEEand Song Guo, Senior Member, IEEE

Abstract—The explosive growth of demands on big data processing imposes a heavy burden on computation, storage, andcommunication in data centers, which hence incurs considerable operational expenditure to data center providers. Therefore, costminimization has become an emergent issue for the upcoming big data era. Different from conventional cloud services, one of themain features of big data services is the tight coupling between data and computation as computation tasks can be conducted onlywhen the corresponding data is available. As a result, three factors, i.e., task assignment, data placement and data movement, deeplyinfluence the operational expenditure of data centers. In this paper, we are motivated to study the cost minimization problem via a jointoptimization of these three factors for big data services in geo-distributed data centers. To describe the task completion time with theconsideration of both data transmission and computation, we propose a two-dimensional Markov chain and derive the average taskcompletion time in closed-form. Furthermore, we model the problem as a mixed-integer non-linear programming (MINLP) and proposean efficient solution to linearize it. The high efficiency of our proposal is validated by extensive simulation based studies.

Index Terms—big data, data flow, data placement, distributed data centers, cost minimization, task assignment

F

1 INTRODUCTION

Data explosion in recent years leads to a rising demandfor big data processing in modern data centers that areusually distributed at different geographic regions, e.g.,Google’s 13 data centers over 8 countries in 4 continents[1]. Big data analysis has shown its great potential inunearthing valuable insights of data to improve decision-making, minimize risk and develop new products andservices. On the other hand, big data has already trans-lated into big price due to its high demand on computa-tion and communication resources [2]. Gartner predictsthat by 2015, 71% of worldwide data center hardwarespending will come from the big data processing, whichwill surpass $126.2 billion. Therefore, it is imperativeto study the cost minimization problem for big dataprocessing in geo-distributed data centers.

Many efforts have been made to lower the computa-tion or communication cost of data centers. Data centerresizing (DCR) has been proposed to reduce the compu-tation cost by adjusting the number of activated serversvia task placement [3]. Based on DCR, some studies haveexplored the geographical distribution nature of datacenters and electricity price heterogeneity to lower theelectricity cost [4]–[6]. Big data service frameworks, e.g.,[7], comprise a distributed file system underneath, whichdistributes data chunks and their replicas across the datacenters for fine-grained load-balancing and high paralleldata access performance. To reduce the communicationcost, a few recent studies make efforts to improve datalocality by placing jobs on the servers where the input

• Lin Gu, Deze Zeng, Peng Li and Song Guo are with University of Aizu,Aizu-wakamatsu, Fukushima, Japan. E-mail: sguo@u-aizu.ac.jp

data reside to avoid remote data loading [7], [8].Although the above solutions have obtained some

positive results, they are far from achieving the cost-efficient big data processing because of the followingweaknesses. First, data locality may result in a wasteof resources. For example, most computation resourceof a server with less popular data may stay idle. Thelow resource utility further causes more servers to beactivated and hence higher operating cost.

Second, the links in networks vary on the transmissionrates and costs according to their unique features [9], e.g.,the distances and physical optical fiber facilities betweendata centers. However, the existing routing strategy a-mong data centers fails to exploit the link diversity ofdata center networks. Due to the storage and compu-tation capacity constraints, not all tasks can be placedonto the same server, on which their correspondingdata reside. It is unavoidable that certain data must bedownloaded from a remote server. In this case, routingstrategy matters on the transmission cost. As indicatedby Jin et al. [10], the transmission cost, e.g., energy,nearly proportional to the number of network link used.The more link used, the higher cost will be incurred.Therefore, it is essential to lower the number of linksused while satisfying all the transmission requirements.

Third, the Quality-of-Service (QoS) of big data taskshas not been considered in existing work. Similar toconventional cloud services, big data applications alsoexhibit Service-Level-Agreement (SLA) between a ser-vice provider and the requesters. To observe SLA, acertain level of QoS, usually in terms of task comple-tion time, shall be guaranteed. The QoS of any cloudcomputing tasks is first determined by where they areplaced and how many computation resources are allocat-

2

ed. Besides, the transmission rate is another influentialfactor since big data tasks are data-centric and the com-putation task cannot proceed until the correspondingdata are available. Existing studies, e.g., [3], on generalcloud computing tasks mainly focus on the computationcapacity constraints, while ignoring the constraints oftransmission rate.

To conquer above weaknesses, we study the costminimization problem for big data processing via jointoptimization of task assignment, data placement, androuting in geo-distributed data centers. Specifically, weconsider the following issues in our joint optimization.Servers are equipped with limited storage and com-putation resources. Each data chunk has a storage re-quirement and will be required by big data tasks. Thedata placement and task assignment are transparent tothe data users with guaranteed QoS. Our objective isto optimize the big data placement, task assignment,routing and DCR such that the overall computation andcommunication cost is minimized. Our main contribu-tions are summarized as follows:

• To our best knowledge, we are the first to considerthe cost minimization problem of big data pro-cessing with joint consideration of data placement,task assignment and data routing. To describe therate-constrained computation and transmission inbig data processing process, we propose a two-dimensional Markov chain and derive the expectedtask completion time in closed form.

• Based on the closed-form expression, we formulatethe cost minimization problem in a form of mixed-integer nonlinear programming (MINLP) to answerthe following questions: 1) how to place these datachunks in the servers, 2) how to distribute tasks ontoservers without violating the resource constraints,and 3) how to resize data centers to achieve theoperation cost minimization goal.

• To deal with the high computational complexity ofsolving MINLP, we linearize it as a mixed-integerlinear programming (MILP) problem, which can besolved using commercial solver. Through extensivenumerical studies, we show the high efficiency ofour proposed joint-optimization based algorithm.

The rest of the paper is organized as follows. Section2 summaries the related work. Section 3 introduces oursystem model. The cost optimization is formulated as anMINLP problem in Section 4 and then it is linearizedin Section 5. The theoretical findings are verified byexperiments in Section 6. Finally, Section 7 concludes ourwork.

2 RELATED WORK

2.1 Server Cost MinimizationLarge-scale data centers have been deployed all overthe world providing services to hundreds of thousandsof users. According to [11], a data center may consistof large numbers of servers and consume megawatts of

power. Millions of dollars on electricity cost have poseda heavy burden on the operating cost to data centerproviders. Therefore, reducing the electricity cost hasreceived significant attention from both academia andindustry [5], [11]–[13]. Among the mechanisms that havebeen proposed so far for data center energy managemen-t, the techniques that attract lots of attention are taskplacement and DCR.

DCR and task placement are usually jointly consideredto match the computing requirement. Liu et al. [4] re-examine the same problem by taking network delay intoconsideration. Fan et al. [12] study power provisioningstrategies on how much computing equipment can besafely and efficiently hosted within a given power bud-get. Rao et al. [3] investigate how to reduce electricitycost by routing user requests to geo-distributed datacenters with accordingly updated sizes that match therequests. Recently, Gao et al. [14] propose the optimalworkload control and balancing by taking account oflatency, energy consumption and electricity prices. Liuet al. [15] reduce electricity cost and environmentalimpact using a holistic approach of workload balancingthat integrates renewable supply, dynamic pricing, andcooling supply.

2.2 Big data Management

To tackle the challenges of effectively managing big data,many proposals have been proposed to improve thestorage and computation process.

The key issue in big data management is reliable andeffective data placement. To achieve this goal, Sathi-amoorthy et al. [16] present a novel family of erasurecodes that are efficiently repairable and offer higherreliability compared to Reed-Solomon codes. They alsoanalytically show that their codes are optimal on an iden-tified tradeoff between locality and minimum distance.Yazd et al. [8] make use of flexibility in the data blockplacement policy to increase energy efficiency in datacenters and propose a scheduling algorithm, which takesinto account energy efficiency in addition to fairness anddata locality properties. Hu et al. [17] propose a mech-anism allowing linked open data to take advantage ofexisting large-scale data stores to meet the requirementson distributed and parallel data processing.

Moreover, how to allocate the computation resourcesto tasks has also drawn much attention. Cohen et al. [18]present new design philosophy, techniques and experi-ence providing a new magnetic, agile and deep data ana-lytics for one of the world’s largest advertising networksat Fox Audience Network, using the Greenplum paralleldatabase system. Kaushik et. al [19] propose a novel,data-centric algorithm to reduce energy costs and withthe guarantee of thermal-reliability of the servers. Chenet al. [20] consider the problem of jointly schedulingall three phases, i.e., map, shuffle and reduce, of theMapReduce process and propose a practical heuristic tocombat the high scheduling complexity.

3

TABLE 1NotationsConstants

Ji The set of servers in data center imi The switch in data center i

w(u,v) The weight of link (u, v)ϕk The size of chunk kλk The task arrival rate for data chunk kP The number of data chunk replicasD The maximum expected response timePj The power consumption of server j

γ(u,v) The transmission rate of link (u, v)

Variablesxj A binary variable indicating if server j is activated

or notyjk A binary variable indicating if chunk k is placed

on server j or notz(u,v)jk A binary variable indicating if link (u, v) is used

for flow for chunk k on server jλjk The request rate for chunk k on server jθjk The CPU usage of chunk k on server jµjk The CPU processing rate of chunk k on server j

f(u,v)jk The flow for chunk k destined to server j through

link (u, v)

2.3 Data Placement

Shachnai et al. [21] investigate how to determine aplacement of Video-on-Demand (VoD) file copies on theservers and the amount of load capacity assigned toeach file copy so as to minimize the communication costwhile ensuring the user experience. Agarwal et al. [22]propose an automated data placement mechanism Volleyfor geo-distributed cloud services with the considerationof WAN bandwidth cost, data center capacity limits,data inter-dependencies, etc. Cloud services make use ofVolley by submitting logs of datacenter requests. Volleyanalyzes the logs using an iterative optimization algo-rithm based on data access patterns and client locations,and outputs migration recommendations back to thecloud service. Cidon et al. [23] invent MinCopysets, adata replication placement scheme that decouples datadistribution and replication to improve the data durabil-ity properties in distributed data centers. Recently, Jin etal. [10] propose a joint optimization scheme that simulta-neously optimizes virtual machine (VM) placement andnetwork flow routing to maximize energy savings.

Existing work on data center cost optimization, bigdata management or data placement mainly focuses onone or two factors. To deal with big data processing ingeo-distributed data centers, we argue that it is essentialto jointly consider data placement, task assignment anddata flow routing in a systematical way.

3 SYSTEM MODEL

In this section, we introduce the system model. For theconvenience of the readers, the major notations used inthis paper are listed in Table 1.

Fig. 1. Data center topology

3.1 Network ModelWe consider a geo-distributed data center topology asshown in Fig. 1, in which all servers of the same datacenter (DC) are connected to their local switch, whiledata centers are connected through switches. There area set I of data centers, and each data center i ∈ Iconsists of a set Ji of servers that are connected toa switch mi ∈ M with a local transmission cost ofCL. In general, the transmission cost CR for inter-datacenter traffic is greater than CL, i.e., CR > CL. Withoutloss of generality, all servers in the network have thesame computation resource and storage capacity, both ofwhich are normalized to one unit. We use J to denotethe set of all severs, i.e., J = J1

∪J2 · · ·

∪J|I|.

The whole system can be modeled as a directed graphG = (N,E). The vertex set N = M

∪J includes the

set M of all switches and the set J of all servers, andE is the directional edge set. All servers are connectedto, and only to, their local switch via intra-data centerlinks while the switches are connected via inter-datacenter links determined by their physical connection.The weight of each link w(u,v), representing the corre-sponding communication cost, can be defined as

w(u,v) =

{CR, if u, v ∈ M,

CL, otherwise.(1)

3.2 Task ModelWe consider big data tasks targeting on data stored ina distributed file system that is built on geo-distributeddata centers. The data are divided into a set K of chunks.Each chunk k ∈ K has the size of ϕk(ϕk ≤ 1), which isnormalized to the server storage capacity. P -way replica[19] is used in our model. That is, for each chunk, thereare exactly P copies stored in the distributed file systemfor resiliency and fault-tolerance.

It has been widely agreed that the tasks arrival at datacenters during a time period can be viewed as a Poissonprocess [9], [24]. In particular, let λk be the average taskarrival rate requesting chunk k. Since these tasks will bedistributed to servers with a fixed probability, the taskarrival in each server can be also regarded as a Poissonprocess. We denote the average arrival rate of task for

4

chunk k on server j as λjk(λjk ≤ 1). When a task isdistributed to a server where its requested data chunkdoes not reside, it needs to wait for the data chunk to betransferred. Each task should be responded in time D.

Moreover, in practical data center management, manytask predication mechanisms based on the historical s-tatistics have been developed and applied to the decisionmaking in data centers [19]. To keep the data centersettings up-to-date, data center operators may makeadjustment according to the task predication period byperiod [3], [14], [15]. This approach is also adopted inthis paper.

4 PROBLEM FORMULATION

In this section, we first present the constraints of dataand task placement, remote data loading, and QoS. Then,we give the complete formulation of the cost minimiza-tion problem in a mixed-integer nonlinear programmingform.

4.1 Constraints of Data and Task Placement

We define a binary variable yjk to denote whether chunkk is placed on server j as follows,

yjk =

{1, if chunk k is placed on server j,

0, Otherwise.(2)

In the distributed file system, we maintain P copiesfor each chunk k ∈ K, which leads to the followingconstraint: ∑

j∈J

yjk = P, ∀k ∈ K. (3)

Furthermore, the data stored in each server j ∈ Jcannot exceed its storage capacity, i.e.,∑

k∈K

yjk · ϕk ≤ 1, ∀j ∈ J. (4)

As for task distribution, the sum rates of task assignedto each server should be equal to the overall rate,

λk =∑j∈J

λjk,∀k ∈ K. (5)

Finally, we define a binary variable xj to denotewhether server j is activated, i.e.,

xj =

{1, if this server is activated,0, otherwise.

(6)

A server shall be activated if there are data chunksplaced onto it or tasks assigned to it. Therefore, we have∑

k∈K yjk +∑

k∈K λjk

K +∑

k∈K λk≤ xj ≤

∑k∈K

yjk+A∑k∈K

λjk,∀j ∈ J,

(7)where A is an arbitrarily large number.

Fig. 2. Two-dimensional Markov Chain

4.2 Constraints of Data Loading

Note that when a data chunk k is required by a server j,it may cause internal and external data transmissions.This routing procedure can be formulated by a flowmodel. All the nodes N in graph G, including the serversand switches, can be divided into three categories:

• Source nodes u(u ∈ J). They are the servers withchunk k stored in it. In this case, the total outletflows to destination server j for chunk k from allsource nodes shall meet the total chunk requirementper time unit as λjk · ϕk.

• Relay nodes mi(mi ∈ M). They receive data flowsfrom source nodes and forward them according tothe routing strategy.

• Destination node j(j ∈ J). When the required chunkis not stored in the destination node, i.e., yjk = 0,it must receive the data flows of chunk k at a rateλjk · ϕk.

Let f(u,v)jk denote the flow over the link (u, v) ∈ E

carrying data of chunk k ∈ K and destined to serverj ∈ J . Then, the constraints on the above three categoriesof nodes can be expressed as follows respectively.

f(u,v)jk ≤ yuk · λk · ϕk, ∀(u, v) ∈ E, u, j ∈ J, k ∈ K (8)∑

(u,v)∈E

f(u,v)jk −

∑(v,w)∈E

f(v,w)jk = 0, ∀v ∈ M, j ∈ J, k ∈ K (9)

∑(u,j)∈E

f(u,j)jk = (1− yjk)λjk · ϕk, ∀j ∈ J, k ∈ K (10)

Note that a non-zero flow f(u,v)jk emitting from server

u only if it keeps a copy of chunk k, i.e., yuk = 1, ascharacterized in (8). The flow conservation is maintainedon each switch as shown in (9). Finally, the destinationreceives all data λk ·ϕk from others only when it does nothold a copy of chunk k, i.e., yik = 0. This is guaranteedby (10).

4.3 Constraints of QoS Satisfaction

Let µjk and γjk be the processing rate and loading ratefor data chunk k on server j, respectively. The processingprocedure then can be described by a two-dimensional

5

Markov chain as Fig. 2, where each state (p, q) representsp pending tasks and q available data chunks.

We let θjk denote the amount of computation resource(e.g., CPU) that chunk k occupies. The processing rate oftasks is proportional to its computation resource usage,i.e.,

µjk = αj · θjk, ∀j ∈ J, k ∈ K, (11)

where αj is a constant relying on the speed of server j.Furthermore, the total computation resource allocated

to all chunks on each server j shall not exceed its totalcomputation resource, i.e.,∑

k∈K

θjk ≤ 1, ∀j ∈ J. (12)

The loading rate of γjk is constrained by the rate onany link (u, v) denoted as γ(u,v), if a non-zero flow f

(u,v)jk

goes through it. This condition can be described by abinary variable z

(u,v)jk as

f(u,v)jk ≤ z

(u,v)jk ≤ Af

(u,v)jk , ∀(u, v) ∈ E, j ∈ J, k ∈ K. (13)

Finally, the constraints on γjk is given as

γjk ≤ γ(u,v) · z(u,v)jk + 1− z(u,v)jk , ∀(u, v) ∈ E, j ∈ J, k ∈ K.

(14)Note that we consider sufficient bandwidth on each

link such that γ(u,v) can be handled as a constant num-ber, which is mainly determined by I/O and switchlatency [25].

By denoting πjk(p, q) as the steady state probabilitythat the Markov chain stays at (p, q), we can describethe transition process by a group of ODEs as follows.According to the transition characteristics, the wholefigure can be divided into three regions.

Region-I: all states in the first line. In Region-I, exceptstate (0, 0), state (p, 0)(p > 1) transits to two neighboringstates (p+ 1, 0) and (p, 1). These can be described as:

π′jk(0, 0) = −λjkπjk(0, 0) + µjkπjk(1, 1), ∀j ∈ J, k ∈ K.

(15)π′jk(p, 0) =− λjk(πjk(p, 0)− πjk(p− 1, 0))

+ µjkπjk(p+ 1, 1)− γπjk(p, 0),∀j ∈ J, k ∈ K.(16)

Region II: all states in the diagonal line except (0, 0). Inthis region, all the pending tasks have already obtainedtheir needed data chunk to proceed. Therefore, eachstate (p, q) in Region-II will transit to (p− 1, q − 1) afterprocessing one data chunk. Then, we have:

π′jk(p, p) =− λjkπjk(p, p) + µjk(πjk(p+ 1, p+ 1)−

πjk(p, p)) + γπjk(p, p− 1), ∀j ∈ J, k ∈ K.(17)

Region-III: all remaining states in the central region.Each state (p, q) in Region-III relies on its three neigh-boring states and also will transit to the other three

neighboring states. As shown in Fig. 2, the transitionrelationship can be written as:

π′jk(p, q) =− λjk(πjk(p, q)− πjk(p− 1, q − 1))

+ µjk(πjk(p+ 1, q + 1)− πjk(p, q))

− γ(πjk(p, q)− πjk(p− 1, q − 1)),

∀j ∈ J, k ∈ K.

(18)

By solving the above ODEs, we can derive the stateprobability πjk(p, q) as:

πjk(p, q) =(λjk)

p(µjk)B−q(γjk)

B−p+q∑Bq=0

∑Bp=0(λjk)p(µjk)B−q(γjk)B−p+q

,

∀j ∈ J, k ∈ K,

(19)

where B is the task buffer size on each server. When Bgoes to infinity, the mean number of tasks for chunk kon server j Tjk is

Tjk = limB→∞

∑Bq=0

∑Bp=0 p(λjk)

p(µjk)B−q(γjk)

B−p+q∑Bq=0

∑Bp=0(λjk)p(µjk)B−q(γjk)B−p+q

,

∀j ∈ J, k ∈ K.(20)

By applying the multivariate l’Hospital’s rule, (20) canbe simplified to

Tjk =λjk

µjkγjk − λjk, ∀j ∈ J, k ∈ K. (21)

According to the Little’s law, the expected delay djkof user requests for chunk k on server j is

djk =Tjk

λjk=

1

µjkγjk − λjk, ∀j ∈ J, k ∈ K. (22)

According to the QoS requirement, i.e., djk ≤ D, wehave

µjkγjk − λjk ≥ ujk

D, ∀j ∈ J, k ∈ K, (23)

where

ujk =

{1, if λjk ̸= 0,0, otherwise.

(24)

The binary variable of ujk can be described by con-straints:

λjk ≤ ujk ≤ Aλjk, ∀j ∈ J, k ∈ K, (25)

where A is an arbitrary large number, because of 0 <λjk < 1 and ujk ∈ {0, 1}.

4.4 An MINLP FormulationThe total energy cost then can be calculated by summingup the cost on each server across all the geo-distributeddata centers and the communication cost, i.e.,

Ctotal =∑j∈J

xj · Pj +∑j∈J

∑k∈K

∑(u,v)∈E

f(u,v)jk · w(u,v), (26)

where Pj is the cost of each activated server j.Our goal is to minimize the total cost by choosing

the best settings of xj , yjk, z(u,v)jk , θjk, λjk and f

(u,v)jk . By

6

summarizing all constraints discussed above, we canformulate this cost minimization as a mixed-integer non-linear programming (MINLP) problem as:

MINLP:min : (26),s.t. : (3) − (5), (7) − (14), (23), (25),

xj , yjk, zjk, ujk ∈ {0, 1},∀j ∈ J, k ∈ K

Note that the above formulation is based on the settingthat the number of replicas for each data chunk is apredetermined constant. If it is a part of the optimization,denoted by an integer variable p, the total cost can befurther minimized by the formulation below.

MINLP-2:min : (26),

s.t. :∑j∈J

yjk = p,∀k ∈ K,

p ≥ 1,

(4), (5), (7) − (14), (23), (25),xj , yjk, zjk, ujk ∈ {0, 1}, ∀j ∈ J, k ∈ K.

5 LINEARIZATION

We observe that the constraints (8) and (10) are nonlineardue to the products of two variables. To linearize theseconstraints, we define a new variable δjk as follows:

δjk = yjkλjk, ∀j ∈ J,∀k ∈ K, (27)

which can be equivalently replaced by the followinglinear constraints:

0 ≤ δjk ≤ λjk,∀j ∈ J,∀k ∈ K, (28)λjk + yjk − 1 ≤ δjk ≤ yjk,∀j ∈ J,∀k ∈ K. (29)

The constraints (8) and (10) can be written in a linearform as:

f(u,v)jk ≤ δukϕk,∀(u, v) ∈ E, u, j ∈ J, k ∈ K, (30)∑

(u,j)∈E

f(u,j)jk = (λjk − δjk) · ϕk,∀j ∈ J, k ∈ K. (31)

We then consider the remaining nonlinear constraints(14) and (23) that can be equivalently written as:

γ(u,v)µjkz(u,v)jk + 1− µjkz

(u,v)jk − λjk ≥ ujk

D,

∀(u, v) ∈ E,∀j ∈ J,∀k ∈ K.(32)

In a similar way, we define a new variable ϵjk as:

ϵ(u,v)jk = µjkz

(u,v)jk , (33)

such that constraint (32) can be written as:

γ(u,v)ϵ(u,v)jk + µjk − ϵ

(u,v)jk − λjk ≥ ujk

D,

∀(u, v) ∈ E,∀j ∈ J,∀k ∈ K.(34)

The constraint (33) can be linearized by:

0 ≤ ϵ(u,v)jk ≤ µjk, ∀(u, v) ∈ E,∀j ∈ J,∀k ∈ K, (35)

µjk + z(u,v)jk − 1 ≤ ϵ

(u,v)jk ≤ z

(u,v)jk , ∀j ∈ J,∀k ∈ K. (36)

Now, we can linearize the MINLP problem into amixed-integer linear programming (MILP) as

MILP:min : (26),s.t. : (3) − (5), (7), (9), (11) − (13),

(25), (28) − (31), (34) − (36),xj , yjk, zjk, ujk ∈ {0, 1}, ∀j ∈ J, k ∈ K.

6 PERFORMANCE EVALUATION

In this section, we present the performance results of ourjoint-optimization algorithm (“Joint”) using the MILPformulation. We also compare it against a separate op-timization scheme algorithm (“Non-joint”), which firstfinds a minimum number of servers to be activated andthe traffic routing scheme using the network flow modelas described in Section 4.2.

In our experiments, we consider |J | = 3 data centers,each of which is with the same number of servers. Theintra- and inter-data center link communication cost areset as CL = 1 and CR = 4, respectively. The cost Pj oneach activated server j is set to 1. The data size, storagerequirement, and task arrival rate are all randomly gen-erated. To solve the MILP problem, commercial solverGurobi [26] is used.

The default settings in our experiments are as follows:each data center with a size 20, the number of datachunks |K| = 10, the task arrival rates λk ∈ [0.01, 5], ∀k ∈K, the number of replicas P = 3, the data chunk sizeϕk ∈ [0.01, 1], ∀k ∈ K, and D = 100. We investigate howvarious parameters affect the overall computation, com-munication and overall cost by varying one parameterin each experiment group.

Fig. 3 shows the server cost, communication cost andoverall cost under different total server numbers varyingfrom 36 to 60. As shown in Fig. 3(a), we can see thatthe server cost always keep constant on any data centersize. As observed from Fig. 3(b), when the total numberof servers increases from 36 to 48, the communicationcosts of both algorithms decrease significantly. This isbecause more tasks and data chunks can be placed inthe same data center when more servers are providedin each data center. Hence, the communication cost isgreatly reduced. However, after the number of serverreaching 48, the communication costs of both algorithmsconverge. The reason is that most tasks and their corre-sponding data chunks can be placed in the same datacenter, or even in the same server. Further increasingthe number of servers will not affect the distributionsof tasks or data chunks any more. Similar results areobserved in Fig. 3(c).

7

35 40 45 50 55 6029

29.5

30

30.5

31

Number of Servers

Ser

ver

Co

st

Joint

Non−joint

(a) Server Cost

35 40 45 50 55 602

4

6

8

10

12

Number of Servers

Com

munic

atio

n C

ost

Joint

Non−joint

(b) Communication Cost

35 40 45 50 55 6032

34

36

38

40

42

Number of Servers

Oper

atio

n C

ost

Joint

Non−joint

(c) Overall Cost

Fig. 3. On the effect of the number of servers

Then, we investigate how the task arrival rate affectsthe cost via varying its value from 29.2 to 43.8. Theevaluation results are shown in Fig. 4. We first noticethat the total cost shows as an increasing function of thetask arrival rates in both algorithms. This is because, toprocess more requests with the guaranteed QoS, morecomputation resources are needed. This leads to anincreasing number of activated servers and hence higherserver cost, as shown in Fig. 4(a). An interesting factnoticed from Fig. 4(a) is that “Joint” algorithm requiressometimes higher server cost than “Non-joint”. This isbecause the first phase of the “Non-joint” algorithmgreedily tries to lower the server cost. However, “Joint”algorithm balances the tradeoff between server cost andcommunication cost such that it incurs much lower com-munication cost and thus better results on the overallcost, compared to the “Non-joint” algorithm, as shown

25 30 35 40 4530

35

40

45

Tasks Arrival Rate

Ser

ver

Cost

Joint

Non−joint

(a) Server Cost

25 30 35 40 450

5

10

15

20

Tasks Arrival RateC

om

munic

atio

n C

ost

Joint

Non−joint

(b) Communication Cost

25 30 35 40 4530

35

40

45

50

55

60

Tasks Arrival Rate

Oper

atio

n C

ost

Joint

Non−joint

(c) Overall Cost

Fig. 4. On the effect of task arrival rate

in Fig. 4(b) and Fig. 4(c), respectively.Fig. 5 illustrates the cost as a function of the total data

chunk size from 8.4 to 19. Larger chunk size leads toactivating more servers with increased server cost asshown in Fig. 5(a). At the same time, more resultingtraffic over the links creates higher communication costas shown in Fig. 5(b). Finally, Fig. 5(c) illustrates theoverall cost as an increasing function of the total datasize and shows that our proposal outperforms “Non-joint” under all settings.

Next we show in Fig. 6 the results when the expectedmaximum response time D increases from 20 to 100.From Fig. 6(a), we can see that the server cost is a non-increasing function of D. The reason is that when thedelay requirement is very small, more servers will beactivated to guarantee the QoS. Therefore, the servercosts of both algorithms decrease as the delay constraint

8

8 10 12 14 16 18 2030

35

40

45

50

55

60

Data Size

Ser

ver

Cost

Joint

Non−joint

(a) Server Cost

8 10 12 14 16 18 202

4

6

8

10

12

14

Data Size

Com

munic

atio

n C

ost

Joint

Non−joint

(b) Communication Cost

8 10 12 14 16 18 2030

40

50

60

70

80

Data Size

Oper

atio

n C

ost

Joint

Non−joint

(c) Overall Cost

Fig. 5. On the effect of data size

increases. A looser QoS requirement also helps find cost-efficient routing strategies as illustrated in Fig. 6(b).Moreover, the advantage of our “Joint” over “Non-joint”can be always observed in Fig. 6(c).

Finally, Fig. 7 investigates the effect of the number ofreplicas for each data chunk, which is set from 1 to 6.An interesting observation from Fig. 7(c) is that the totalcost first decreases and then increases with the increasingnumber of replicas. Initially, when the replica numberincreases from 1 to 4, a limited number of activatedservers are always enough for task processing, as shownin Fig. 7(a). Meanwhile, it improves the possibility thattask and its required data chunk are placed on thesame server. This will reduce the communication cost, asshown in Fig. 7(b). When the replica number becomeslarge, no further benefits to communication cost willbe obtained while more servers must be activated only

20 40 60 80 10030

30.5

31

31.5

32

Delay

Ser

ver

Cost

Joint

Non−joint

(a) Server Cost

20 40 60 80 1002

4

6

8

10

12

14

DelayC

om

mu

nic

atio

n C

ost

Joint

Non−joint

(b) Communication Cost

20 40 60 80 10030

35

40

45

Delay

Op

erat

ion

Co

st

Joint

Non−joint

(c) Overall Cost

Fig. 6. On the effect of expected task completion delay

for the purpose of providing enough storage resources.In this case, the server and hence the overall costsshall be increased, as shown in Fig. 7(a) and Fig. 7(c),respectively.

Our discovery that the optimal number of chunkreplicas is equal to 4 under the network setting aboveis verified one more time by solving the formulationgiven in Section 4.4 that is to minimize the number ofreplicas with the minimum total cost. Additional resultsare given under different settings via varying the taskarrival rate and chunk size in the ranges of [0.1, λU ] and[ϕL, 1.0], respectively, where a number of combinationsof (λU , ϕL) are shown in Fig. 8. We observe that theoptimal number of replica a non-decreasing function ofthe task arrival rate under the same chunk size while anon-increasing function of the data chunk size under thesame task arrival rate.

9

1 2 3 4 5 630

35

40

45

50

55

Number of Replica

Ser

ver

Cost

Joint

Non−joint

(a) Server Cost

1 2 3 4 5 60

5

10

15

20

25

30

Number of Replica

Com

mun

icat

ion

Cost

Joint

Non−joint

(b) Communication Cost

1 2 3 4 5 630

35

40

45

50

55

60

Number of Replica

Oper

atio

n C

ost

Joint

Non−joint

(c) Overall Cost

Fig. 7. On the effect of the number of replica

7 CONCLUSION

In this paper, we jointly study the data placement, taskassignment, data center resizing and routing to minimizethe overall operational cost in large-scale geo-distributeddata centers for big data applications. We first character-ize the data processing process using a two-dimensionalMarkov chain and derive the expected completion timein closed-form, based on which the joint optimization isformulated as an MINLP problem. To tackle the highcomputational complexity of solving our MINLP, welinearize it into an MILP problem. Through extensive ex-periments, we show that our joint-optimization solutionhas substantial advantage over the approach by two-stepseparate optimization. Several interesting phenomenaare also observed from the experimental results.

(3,0.1) (4,0.1) (5,0.1) (3,0.8) (4,0.8) (5,0.8)0

0.5

1

1.5

2

2.5

3

3.5

4

Arrival Rate & Chunk Size

Opti

mal

Num

ber

of

Rep

lica

Fig. 8. Optimal number of replica

REFERENCES

[1] “Data Center Locations,” http://www.google.com/about/datacenters/inside/locations/index.html.

[2] R. Raghavendra, P. Ranganathan, V. Talwar, Z. Wang, and X. Zhu,“No “Power” Struggles: Coordinated Multi-level Power Manage-ment for the Data Center,” in Proceedings of the 13th InternationalConference on Architectural Support for Programming Languages andOperating Systems (ASPLOS). ACM, 2008, pp. 48–59.

[3] L. Rao, X. Liu, L. Xie, and W. Liu, “Minimizing Electricity Cost:Optimization of Distributed Internet Data Centers in a Multi-Electricity-Market Environment,” in Proceedings of the 29th Interna-tional Conference on Computer Communications (INFOCOM). IEEE,2010, pp. 1–9.

[4] Z. Liu, M. Lin, A. Wierman, S. H. Low, and L. L. Andrew,“Greening Geographical Load Balancing,” in Proceedings of In-ternational Conference on Measurement and Modeling of ComputerSystems (SIGMETRICS). ACM, 2011, pp. 233–244.

[5] R. Urgaonkar, B. Urgaonkar, M. J. Neely, and A. Sivasubrama-niam, “Optimal Power Cost Management Using Stored Energyin Data Centers,” in Proceedings of International Conference onMeasurement and Modeling of Computer Systems (SIGMETRICS).ACM, 2011, pp. 221–232.

[6] B. L. Hong Xu, Chen Feng, “Temperature Aware WorkloadManagement in Geo-distributed Datacenters,” in Proceedings ofInternational Conference on Measurement and Modeling of ComputerSystems (SIGMETRICS). ACM, 2013, pp. 33–36.

[7] J. Dean and S. Ghemawat, “Mapreduce: simplified data process-ing on large clusters,” Communications of the ACM, vol. 51, no. 1,pp. 107–113, 2008.

[8] S. A. Yazd, S. Venkatesan, and N. Mittal, “Boosting energy ef-ficiency with mirrored data block replication policy and energyscheduler,” SIGOPS Oper. Syst. Rev., vol. 47, no. 2, pp. 33–40, 2013.

[9] I. Marshall and C. Roadknight, “Linking cache performance touser behaviour,” Computer Networks and ISDN Systems, vol. 30,no. 223, pp. 2123 – 2130, 1998.

[10] H. Jin, T. Cheocherngngarn, D. Levy, A. Smith, D. Pan, J. Liu,and N. Pissinou, “Joint Host-Network Optimization for Energy-Efficient Data Center Networking,” in Proceedings of the 27thInternational Symposium on Parallel Distributed Processing (IPDPS),2013, pp. 623–634.

[11] A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, and B. Maggs,“Cutting the Electric Bill for Internet-scale Systems,” in Proceed-ings of the ACM Special Interest Group on Data Communication(SIGCOMM). ACM, 2009, pp. 123–134.

[12] X. Fan, W.-D. Weber, and L. A. Barroso, “Power Provisioning forA Warehouse-sized Computer,” in Proceedings of the 34th AnnualInternational Symposium on Computer Architecture (ISCA). ACM,2007, pp. 13–23.

[13] S. Govindan, A. Sivasubramaniam, and B. Urgaonkar, “Benefitsand Limitations of Tapping Into Stored Energy for Datacenters,”in Proceedings of the 38th Annual International Symposium on Com-puter Architecture (ISCA). ACM, 2011, pp. 341–352.

10

[14] P. X. Gao, A. R. Curtis, B. Wong, and S. Keshav, “It’s Not EasyBeing Green,” in Proceedings of the ACM Special Interest Group onData Communication (SIGCOMM). ACM, 2012, pp. 211–222.

[15] Z. Liu, Y. Chen, C. Bash, A. Wierman, D. Gmach, Z. Wang,M. Marwah, and C. Hyser, “Renewable and Cooling Aware Work-load Management for Sustainable Data Centers,” in Proceedings ofInternational Conference on Measurement and Modeling of ComputerSystems (SIGMETRICS). ACM, 2012, pp. 175–186.

[16] M. Sathiamoorthy, M. Asteris, D. Papailiopoulos, A. G. Dimakis,R. Vadali, S. Chen, and D. Borthakur, “Xoring elephants: novelerasure codes for big data,” in Proceedings of the 39th internationalconference on Very Large Data Bases, ser. PVLDB’13. VLDBEndowment, 2013, pp. 325–336.

[17] B. Hu, N. Carvalho, L. Laera, and T. Matsutsuka, “Towards biglinked data: a large-scale, distributed semantic data storage,”in Proceedings of the 14th International Conference on InformationIntegration and Web-based Applications & Services, ser. IIWAS ’12.ACM, 2012, pp. 167–176.

[18] J. Cohen, B. Dolan, M. Dunlap, J. M. Hellerstein, and C. Welton,“Mad skills: new analysis practices for big data,” Proc. VLDBEndow., vol. 2, no. 2, pp. 1481–1492, 2009.

[19] R. Kaushik and K. Nahrstedt, “T*: A data-centric cooling energycosts reduction approach for Big Data analytics cloud,” in 2012 In-ternational Conference for High Performance Computing, Networking,Storage and Analysis (SC), 2012, pp. 1–11.

[20] F. Chen, M. Kodialam, and T. V. Lakshman, “Joint scheduling ofprocessing and shuffle phases in mapreduce systems,” in Proceed-ings of the 29th International Conference on Computer Communications(INFOCOM). IEEE, 2012, pp. 1143–1151.

[21] H. Shachnai, G. Tamir, and T. Tamir, “Minimal cost reconfigu-ration of data placement in a storage area network,” TheoreticalComputer Science, vol. 460, pp. 42–53, 2012.

[22] S. Agarwal, J. Dunagan, N. Jain, S. Saroiu, A. Wolman, and H. B-hogan, “Volley: Automated Data Placement for Geo-DistributedCloud Services,” in The 7th USENIX Symposium on NetworkedSystems Design and Implementation (NSDI), 2010, pp. 17–32.

[23] A. Cidon, R. Stutsman, S. Rumble, S. Katti, J. Ousterhout, andM. Rosenblum, “MinCopysets: Derandomizing Replication InCloud Storage,” in The 10th USENIX Symposium on NetworkedSystems Design and Implementation (NSDI), 2013.

[24] S. Gunduz and M. Ozsu, “A poisson model for user accesses toweb pages,” in Computer and Information Sciences - ISCIS 2003, ser.Lecture Notes in Computer Science. Springer Berlin Heidelberg,2003, vol. 2869, pp. 332–339.

[25] L. Kleinrock, “The latency/bandwidth tradeoff in gigabit net-works,” Communications Magazine, IEEE, vol. 30, no. 4, pp. 36–40,1992.

[26] “Gurobi,” www.gurobi.com.

Lin Gu Lin Gu received her M.S. degreesin computer science from University of Aizu,Fukushima, Japan, in 2011. She is currently pur-suing her Ph.D. in University of Aizu, Japan. Shereceived her B.S. degree in computer scienceand technology from School of Computer Sci-ence and Technology, Huazhong University ofScience and Technology, China in 2009. She isa student member of IEEE. Her current researchinterests include cloud computing, big data andsoftware-defined networking.

Deze Zeng received his Ph.D. and M.S. degreesin computer science from University of Aizu,Aizu-Wakamatsu, Japan, in 2013 and 2009, re-spectively. He is currently a research assistantin University of Aizu, Japan. He received hisB.S. degree from School of Computer Scienceand Technology, Huazhong University of Sci-ence and Technology, China in 2007. He is amember of IEEE. His current research interestsinclude cloud computing, networking protocoldesign and analysis, with a special emphasis on

delay-tolerant networks and wireless sensor networks.

Peng Li received his BS degree from HuazhongUniversity of Science and Technology, China, in2007, the MS and PhD degrees from the Univer-sity of Aizu, Japan, in 2009 and 2012, respec-tively. He is currently a Postdoctoral Researcherin the University of Aizu, Japan. His researchinterests include networking modeling, cross-layer optimization, network coding, cooperativecommunications, cloud computing, smart grid,performance evaluation of wireless and mobilenetworks for reliable, energy-efficient, and cost-

effective communications.

Song Guo (M’02-SM’11) received the PhD de-gree in computer science from the Universityof Ottawa, Canada in 2005. He is currently aFull Professor at School of Computer Scienceand Engineering, the University of Aizu, Japan.His research interests are mainly in the areasof protocol design and performance analysis forreliable, energy-efficient, and cost effective com-munications in wireless networks. He receivedthe Best Paper Awards at ACM Conference onUbiquitous Information Management and Com-

munication 2014, IEEE Conference on Computational Science and En-gineering 2011, and IEEE Conference on High Performance Computingand Communications 2008. Dr. Guo currently serves as Associate Editorof the IEEE Transactions on Parallel and Distributed Systems. He is inthe editorial boards of ACM/Springer Wireless Networks, Wireless Com-munications and Mobile Computing, International Journal of DistributedSensor Networks, International Journal of Communication Networksand Information Security, and others. He has also been in organizingcommittee of many international conferences, including serving as aGeneral Chair of MobiQuitous 2013. He is a senior member of the IEEEand the ACM.