This document may differ from the final, published version of the research and has been made available online in accordance with publisher policies. To read and/or cite from the published version of the research, please visit the publisher’s website (a subscription may be required.)
An Optimal Task Placement Strategy inGeo-Distributed Data Centers involvingRenewable EnergyRAN WANG1, 2, (Member, IEEE), YIWEN LU1, KUN ZHU1, 2, (Member, IEEE), JIE HAO1, 2,(Member, IEEE), PING WANG3, (Senior Member, IEEE), and YUE CAO4, (Member, IEEE)1College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China(e-mail: {wangran, luyiwen, zhukun, haojie}@nuaa.edu.cn)
2Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing, China3Department of Electrical Engineering and Computer Science, York University, Canada (e-mail: [email protected])4Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne, UK (e-mail: [email protected])
This work was supported by the Fundamental Research Funds for the Central Universities, NO. NS2017072.
ABSTRACT Nowadays, modern data centers are seeking for importing renewable energy together withconventional energy in order to be more environment-friendly and to reduce operation expenditures.Meanwhile, considering the fact that electricity prices and renewable energy generations are diverse intime and geography, a task scheduling strategy should be designed to ensure the efficient and economicoperations of data centers. In this paper, an optimal task placement strategy is presented for geo-distributeddata centers powered by mixed renewable and conventional energies with dynamic voltage and frequencyscaling (DVFS) technique. We aim at minimizing the total electricity cost and making full use of therenewable energy so as to construct green and economic data centers. The optimal task placement problem isformulated as a mixed integer nonlinear problem (MINLP), in which the quality-of-service (QoS) constraintis restricted by an M/G/1 queuing model. To tackle the complexity of the MINLP, we first transform itinto a tractable form, then develop an optimal sever activation configuration (SAC) and task placementalgorithm to solve it. The proposed algorithm can obtain the global optimal solution of the electricityminimization problem and meanwhile dramatically reduce the complexity of the problem solving. Finally,evaluations based on real-world traces exhibit impacts of different system parameters on the electricity costand sever activation configurations, which prove the superiority of our proposed algorithm and provide ussome illuminations on how to build cost-effective and eco-friendly data centers.
INDEX TERMS data centers; dynamic voltage and frequency scaling (DVFS); renewable energy; severactivation configuration (SAC); task placement.
I. INTRODUCTION
Data centers are always working in 24 hours day andnight without shutting down, consuming enormous electricitywhich mainly comes from conventional energy such as coal,oil and natural gas, thus causing serious air pollution tothe environment [1, 2]. There is an estimation that a datacenter with 5 × 104 servers may use over 100 million KWhper year [3], as much as the urban energy consumption for105 households. Under such circumstance, renewable energypowered data centers have attracted more attentions in recentyears due to their low costs and environment-friendly prop-erties. While renewable energy offers a cheaper and cleanerelectricity supply, the integration of climate-dependent re-
newable energy into data centers also imposes great opera-tion challenges because of its high inter-temporal variationand limited predictability. In addition, electricity price andrenewable energy generation (REG) are diverse in time andgeography domains, hence an optimal task placement strat-egy should be properly designed to dynamically coordinatethe renewable energy generation and task placement of datacenters so that the energy expenditure for operating the datacenters is minimized while tasks can be executed in time.In recent years, a power management technique, namelydynamic voltage and frequency scaling (DVFS) is employedto dynamically adjust server’s operating voltage and fre-quency. For example, if user requests flood into front por-
VOLUME *, * 1
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
tals, servers should improve operating frequencies to processmore user requests within a short time. If user requests arenot large, servers can operate at low frequencies for energyconservation. The DVFS technique certainly enables moreflexible task scheduling; meanwhile it will further complicatethe task placement process. As a consequence, optimal taskplacement in renewable energy powered geo-distributed datacenters considering DVFS technique becomes a practical andimportant research problem.
A. RELATED WORKSome research works have studied task scheduling prob-lem in data centers. For pure task placement problem, in[4], H. Xu et al. propose to make workload managementtemperature aware. The workload management problem isformulated as a joint optimization of request routing forinteractive workloads and capacity allocation for batch work-loads. Reference [5] considered the stochastic multi-stage jobscheduling problem on salable resources in data centers tomaximize the utilization of rented VMs over a certain periodof time. X. Zhao et al. in [6] present a model-predictivecontrol (MPC) based scheduling strategy called ThermoRingto reduce cooling costs in data centers. ThermoRing copeswith thermal emergencies by controlling task allocationsto computing nodes. Mateusz et al. in [7] target an on-line scheduling problem of work-flows consisting of inter-related tasks in a data center. A novel methodology namedMinimum Dependencies Energy-efficient Directed AcyclicGraphs scheduling is then developed. In reference [8], Y.Wang et al. build an optimization framework of an interactionsystem of the smart power grid, jointly accounting for theservice request dispatch and routing problem in the datacenter with the power flow analysis in power grid. Literature[9] proposes a temporal task scheduling algorithm (TTSA)to effectively dispatch all arriving work loads to private datacenters and public clouds. In each iteration of TTSA, thecost minimization problem is modeled as a mixed integerlinear program (MILP). L. Yu et al. [10] optimize the prob-lem concerning joint workload and battery scheduling withheterogeneous service delay guarantees for data centers. Anonline operation algorithm to solve the problem based onLyapunov optimization technique is then designed.
As for task scheduling in renewable energy powered datacenters, N. Hogade et al. in [11] design techniques forgeographical load distribution that will minimize the energyexpenditure for executing incoming tasks considering manyaspects of the overall system. In [12], the authors proposea novel analytical model to calculate profit in large datacenters without and with behind-themeter renewable powergeneration. Then the derived profit model is adlpted to de-velop an optimization-based profit maximization strategy fordata centers. A robust workload and energy managementframework for sustainable data centers is developed in [13].To deal with the uncertainty from renewable energy gener-ation, the resource allocation task is formulated as a robustoptimization problem that minimizes the worst-case net cost.
S. Chen et al. in [14] build a comprehensive framework cov-ering the costs of server power, cooling power, and hardwaremaintenance. A joint optimization of the costs of electricityand server maintenance is then introduced. A. N. Toosi et al.in [15] propose a framework for load balancing of web appli-cation requests in geo-distributed data centers based on therenewable energy in each region. Another work can be foundin [16], where a unified management approach is proposedallowing data centers to adaptively respond to intermittentavailability of renewables under long-term quality-of-service(QoS) requirements.
For task placement with DVFS technique, a common wayis to operate servers at high frequency when processing largesize of user requests and operate servers at low frequencywhen their workload is small. Note that high frequencyusually leads to high energy consumption and low frequencymeans long execution time, thus the electricity cost mini-mization and QoS requirements should be balanced. L. Guet al. in [17] design an iterative searching algorithm to solvethe complex user requests allocation problem. S. Wang etal. in [18] propose a DVFS-based task model describing theQoS requirements of tasks, without the prior knowledge ofexecution time of tasks, and then transform the task schedul-ing problem into minimizing the total energy consumptionratio. Z. Tang et al. in [19] propose a DVFS-enabled energy-efficient work-flow task scheduling algorithm by merging therelatively inefficient processors through reclaiming the slacktime, which could make use of the slack time recurrently. H.Lei et al. in [20] built a four-objective framework to optimizeover the utilization of renewable energy, completion time oftasks, total energy consumption and processing rate of tasks.An enhanced multi-objective co-evolutionary algorithm isproposed to solve the problem. C. Wu et al. in [21] propose ascheduling algorithm for the cloud data center with DVFStechnique, which aimed at efficiently increasing resourceutilization hence decreasing the energy consumption for pro-cessing jobs. Readers interested in task scheduling problemin data centers can refer to surveys [22] and [23] for a morecomprehensive understanding.
B. MAIN CONTRIBUTIONSThe above papers [4]-[23] optimized over the operations ofdata centers either with no DVFS-enabled servers where themain consideration was the number of activated servers, orwith no renewable energy imported whose goal was to mini-mize the total energy consumption. Most literature normallymodeled the system from the micro viewpoint. To conquerthese weaknesses, we investigate the task placement problemin geo-distributed data centers considering QoS requirementsand DVFS technique, aiming at minimizing the electricitycost and making full use of renewable energy. The maincontributions of this paper are summarized as follows:
• We describe the dynamic task flow in each server asan M/G/1 queue which is then adopted to formulatethe QoS constraint in data centers. To the best of ourknowledge, this is the first time that the M/G/1 queue
2 VOLUME *, *
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
is employed to model task flow in renewable energypowered data centers.
• We formulate the optimal DVFS based task place-ment problem into a mixed integer nonlinear problem(MINLP) to determine the distribution flow of userrequests to geo-distributed data centers. Factors such asthe REG and electricity price, the number of activatedservers in each data center, the amount of user requestseach data center processes, the amount of conventionalenergy each data center consumes and at what frequencylevel each server operates are jointly considered.
• To deal with the high complexity and non-convexity ofMINLP, we first transform the original problem into anequivalent but less complicated one. Then an optimalserver activation configuration (SAC) and task place-ment algorithm is developed to efficiently solve thetransformed problem in polynomial time, where a globaloptimal solution is achieved.
• Numerical results based on real-world traces evaluatethe impacts of different parameters on electricity costand server operating frequencies, providing some in-sights on how to design task placement policies indata centers. Additionally, the proposed task placementstrategy can reduce the electricity cost considerably,providing us some insights on how to build economicand sustainable data centers.
The reminder of this paper is organized as follows: Sec-tion II introduces the particulars of the system model. InSection III, we show the mathematical formulation of theinitial optimal task placement problem and then transformit into an easier-solving one, which drastically decreases thecomplexity of the original problem. In Section IV, an optimalSAC and task placement strategy is designed to solve therelaxed problem. Simulation results based on real-world dataare presented in Section V. Finally we conclude the paper inSection VI.
II. SYSTEM MODELIn this section, the particulars of the system operation arepresented in details below.
A. SYSTEM ARCHITECTURE OF GEO-DISTRIBUTEDDATA CENTERSGenerally, we model over a system with N geo-distributeddata centers and M front portals deployed on different sites.For easier understanding, a data center system composed of 3front portals and 4 data centers powered by mixed renewableand conventional energies is illustrated in Fig. 1, i.e., M = 3and N = 4. Each front portal is responsible for collectingsurrounding user requests and delivering them to appropriatedata centers. A data center usually consists of great numberof servers, with Sn and Prn denoting server number andelectricity price at data center n, respectively. Note that amulti-electricity market is considered in this paper, whereelectricity price presents regional diversities.
B. DYNAMIC VOLTAGE AND FREQUENCY SCALINGTECHNIQUEDVFS is essentially a power-saving technique, allowing dy-namic adjustment of working frequency of a microproces-sor via regulating the supplied voltage. As we all know,higher supplied voltage normally indicates higher frequencyand larger power consumption meanwhile. Under such case,DVFS technique is utilized in data centers to set reasonableoperating voltage and clock frequency based on the actualpower consumption of the chip at the time, which ensuressufficient but not exceeding power supply. Generally, powerconsumption P is a function of the supplied voltage v and theworking frequency f in the range of fmin to fmax, with fminand fmax denoting the minimum and maximum frequency,respectively. Thus,
P = Bv2f + Pstatic (1)
where B is a coefficient related to different processors andPstatic is the static power consumption independent of f ,which is mainly caused by leakage currents in devices andcircuits. Based on the existing research,Pstatic can be viewedas a constant occupied 10% − 60% of the maximum powerconsumption according to different architectures and tech-nologies. In general, the relationship between the suppliedvoltage and operating frequency is v ∝ fβ , where β is alwaysset as 1. Let α = 2β, then we have
P = Bfα+1 + Pstatic. (2)
C. WORKLOAD MODELIn this paper, we adopt the widely accepted assumption thatuser requests arrive at front portals in a Poisson process [24].We denote the user request arrival rate at front portal m asλm. As soon as user requests arrive at front portals, theywill be distributed to processing servers in geo-distributeddata centers with a predetermined probability, hence the userrequest arrival in a server can be also viewed as a Poissonprocess. Denote λsmn(s ∈ [1, Sn]) as the request’s averagearrival rate at server s in data center n from front portal m.
In modern data centers, management controllers are incharge of the task dispatching to servers, where a local cacheis equipped to buffer the waiting tasks. Without loss of gen-erality, the processing time of a task satisfies a general dis-tribution if the server frequency is given. Thus the dynamicstate of a server’s task queue in data center networks can bemodeled as an M/G/1 queue which has already been adoptedby the previous literature [25, 26]. In addition, the service ratein a DVFS-enabled server is proportional to its frequency,i.e., µsn = rfsn, where fsn is the operating frequency of servers in data center n and r is a scaling factor.
III. PROBLEM FORMULATIONA. THE POWER CONSUMPTION CONSTRAINTLet P sn denote the power consumption of server s in datacenter n. P sn ≡ 0 if a server is deactivated. Otherwise, P sn
VOLUME *, * 3
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
1
2
3
Users
Data centers
Front
portals
Renewable energy
Conventional energy
Energy flow
Request flow
FIGURE 1: An illustration of geo-distributed data centers
is a joint function of processing frequency fsn and usage ρsnwhen it is activated, i.e.,
P sn = B(fsn)α+1ρsn + Pstatic,
where ρsn is the probability that server s in data center n isbusy. According to the M/G/1 queuing theory, we have
ρsn =λsnrfsn
,
where
λsn =
M∑m=1
λsmn
is the total requests distributed to server s in data center nfrom M front portals.
A binary variable Y sn ∈ {0, 1} is used to denote whethera server is activated or not, i.e., Y sn = 1 if it is activated andY sn = 0 otherwise, thus
P sn = Y sn (B(fsn)αλ
sn
r+ Pstatic).
B. THE POWER SUPPLY AND DEMAND BALANCECONSTRAINTThe power demand Ds
n of server s in data center n should beequal to its power consumption P sn as we assume the powerusage effectiveness (PUE) is 1 in ideal status for convenience,i.e.,Ds
n = P sn = Y sn (B(fsn)α λ
sn
r +Pstatic). A data center canobtain electricity from both public power grid and its ownrenewable energy plants. Denote the electricity bought frompublic power grid in server s as Csn and REG in data centern as Rn. In order to satisfy all the user requests, we have the
requirement that the electricity supply should not be less thanthe demand, i.e.,
(
Sn∑s=1
Csn) +Rn ≥Sn∑s=1
Dsn.
C. THE WORKLOAD BALANCE CONSTRAINT
User requests reach the front portalm at a rate λm. Then theyare allocated to servers for executing. Requests dispatched todata center n are the summation of requests executed by allservers in it, i.e.,
∑Sn
s=1 λsn. Therefore, total requests to be
executed in all servers should be equal to those arriving at allfront portals, i.e.,
M∑m=1
λm =
N∑n=1
Sn∑s=1
λsn.
D. THE QOS CONSTRAINT
As aforementioned, the request processing procedure ofserver s in data center n is regarded as an M/G/1 queuingmodel with mean arrival and service rate as λsn and µsn,respectively. In the steady state, the expected delay tsn at eachserver is the summation of average service time xsn = 1
µsn
and
average waiting time wsn =λsn·xs
n2
2(1−ρsn)of the tasks that queue
at server s in data center n [27], i.e.,
tsn = xsn + wsn.
While tasks’ average service time xsn is equal to their averagesize szsn divided by the server’s computing speed cssn, i.e.,xsn =
szsncssn
= 1µsn
in which cssn = kµsn, thus k = szsn =
4 VOLUME *, *
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
E(szsn) is a known factor indicating the expectation of tasksize. Hence we have
tsn =Y snrfsn
+λsnsz
sn2
2(1− λsn
rfsn)(krfsn)
2,
where szsn2 = V (szsn) + [E(szsn)]2. V (szsn) is the variance
of task size which is easy to obtain. To guarantee the QoSrequirement, delay tsn at any activated server should not vio-late the maximum delay T which gives rise to the followingconstraint
Y snrfsn
+λsnsz
sn2
2(1− λsn
rfsn)(krfsn)
2≤ T.
E. AN MINLP FORMULATIONNote that each server in data center n has the same electricityprice Prn, the overall electricity cost EC can be obtained bysumming the conventional electricity cost derived from allservers across all geo-distributed data centers, i.e.,
EC =
N∑n=1
Sn∑s=1
CsnPrn.
Taking all constraints presented above, the optimal taskplacement problem with electricity cost minimization objec-tive is formulated as an MINLP where Y sn , fsn, λsn and Csn aredecision variables. Hence we have
minY sn ,f
sn,λ
sn,C
sn
N∑n=1
Sn∑s=1
CsnPrn (3)
s.t.M∑m=1
λm =
N∑n=1
Sn∑s=1
λsn, (4)
Y snrfsn
+λsnsz
sn2
2(1− λsn
rfsn)(krfsn)
2≤ T, (5)
(
Sn∑s=1
Csn) +Rn ≥Sn∑s=1
Dsn, (6)
Dsn = Y sn (B(fsn)
αλsn
r+ Pstatic), (7)
0 ≤ fsn ≤ fmax, Y sn ∈ {0, 1}. (8)
Due to the high complexity of the problem (time complex-ity is O((2Sn)N )), the following proposition is proposed tosimplify the original MINLP.
Proposition 1. For the given amount of requests, the mini-mum energy consumption in a data center is attained whenuser requests are uniformly delivered to all activated servers.
Proof. Note that the energy consumption function (3) isincreasing with respect to working frequency fsn, it’s reason-able to have
fsn(λsn) =
k(Tλsn + 1)−√k2(Tλsn − 1)2 + 2Tλsnsz
sn2
2Tkr
which is derived from inequation (5) for energy saving whileensuring the QoS requirements. Thus, the minimum powerdemand Ds
n can be rewritten as a function of λsn. Let Sanbe the number of activated servers in data center n, thetotal energy demand can be calculated as
∑Sans=1D
sn(λ
sn).
Since Dsn(λ
sn) is strictly convex in range (0,+∞), applying
Jensen’s inequality we have∑Sa
ns=1D
sn(λ
sn) ≥ Ds
n(∑Sa
ns=1 λ
sn
San
)·San, where equality holds if and only if λ1n = λ2n = ... = λ
Sann .
Therefore, it is proved that the minimum energy consumptionof a data center is achieved via uniformly scheduling allrequests to all activated servers.
Denote hn =∑Sa
ns=1 λ
sn as the requests allocated to data
center n. According to Proposition 1, the minimum energyconsumption in data center n for the given San is equivalentto
It’s obvious to observe that the complexity of the new formu-lation is O((Sn + 1)N ), which is much smaller than that ofthe former one with O((2Sn)N ), owing to two new variablesintroduced which reduce the solution space significantly.
IV. ALGORITHM DESIGNIn this section, we design efficient algorithms to solveproblem (11)-(15). Based on the previous analysis, for thegiven San, problem (11)-(15) is convex and can be solved inpolynomial time. Under such circumstance, a double-casealgorithm will be designed to solve it. The correspondingsolution SAC = [Sa1 , S
a2 , ..., S
an] is named as the server
activation configuration. According to the system design, theSAC is composed of two parts as shown in Fig. 2: SACRErepresenting the number of activated servers powered byrenewable energy (RE) and SACCE representing the numberof activated servers powered by conventional energy (CE).
The optimal SAC and task placement strategy is shown inAlgorithm 1 aiming at solving problem (11)-(15) which fur-ther includes two cases. In Algorithm 1, all data centers are
VOLUME *, * 5
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
activated deactivated
FIGURE 2: An illustration of server activation configuration
sorted by the descending order of REG first and we calculatethe maximum number of activated servers Sa,REn max andcomputation capacity hREn max (i.e. the maximum requeststhat can be processed) of each data center powered only byrenewable energy (described in lines 2-4). Then we comparetotal user requests
∑Mm=1 λm arriving at front portals with
the summation of all data centers’ maximum computationcapacity
∑Nn=1 h
REn max. If the former is smaller than the
latter, all data centers can be powered only by renewableenergy to meet all user requests which is called Case I.Otherwise, conventional energy need to be imported to datacenters which we name such case as Case II.
Algorithm 1 Optimal SAC and Task Placement StrategyInput: M , N , λm, Prn, Sn, fmin, fmax
Output: SAC, EC1: Sort data centers in descending order of REG2: for n = 1 to N do3: {Sa,RE
n max, hREn max} = argmax hRE
n s.t.Dn(S
a,REn , hRE
n ) = Rn, 0 ≤ Sa,REn ≤ Sn, 0 ≤ hRE
n ≤∑Mm=1 λm, fmin ≤ fs
n ≤ fmax
4: end for5: if
∑Mm=1 λm ≤
∑Nn=1 h
REn max then
6: Case I: Data centers are powered by RE.7: else8: Case II: Data centers are powered by RE and CE.9: end if
In Case I (Algorithm 2), we compare all user requests∑Mm=1 λm with total maximum requests htotal that data
centers can process from one with the most REG, then theoptimal SAC set and minimum number of activated datacenters Nmin is obtained.
In Case II (Algorithm 3), some data centers have to bepowered by conventional energy to process the user requests.First, all data centers are sorted in the ascending order bythe electricity prices and we obtain initial SAC by activat-ing the remaining servers from data center with the lowestelectricity price, and the computation capacity hCEn min ofeach data center with minimum energy consumption can becalculated in line 5. Then we compare the remaining userrequests hr with each data center’s minimum computationcapacity hCEn min. If the former is smaller than the latter, the
Algorithm 2 Case I: Data centers powered by RE1: htotal = 0, Nmin = N2: for n = 1 to N do3: htotal = htotal + hRE
n max
4: San = Sa,RE
n max
5: if∑M
m=1 λm ≤ htotal
6: Nmin = n7: break8: end if9: end for
10: SAC = {San, ∀n = [1, Nmin]}
Algorithm 3 Case II: Data centers powered by RE and CE
1: hr =∑M
m=1 λm −∑N
n=1 hREn max, Nmax = N
2: Sort data centers in ascending order by the electricity prices3: for n = 1 to N do4: Sa
n = Sn
5: hCEn min = argmin Dn(Sn − Sa,RE
n max, hCEn ) s.t. 0 ≤
hCEn ≤ hr, fmin ≤ fs
n ≤ fmax
6: if hr ≤ hCEn min
7: Nmax = n8: break9: end if
10: hr = hr − hCEn min
11: end for12: SAC = {Sa
n, ∀n = [1, n]}13: {hn, EC} ← solve problem (11)-(15) under SAC14: for n = Nmax to 1 do15: Sa
ns = 0, Sane = Sa
n
16: while Sans ≤ Sa
ne do17: SAC
′= {Sa
n|San = bS
ans
+Sane
2c}
18: {h′n, EC
′} ← solve problem (11)-(15) under SAC
′
19: if EC′< EC then
20: EC = EC′
21: Sane = Sa
n − 122: else23: Sa
ns = San + 1
24: end if25: end while26: end for
remaining user requests can be processed by this data centerand the maximum number of activated data centers Nmax iscalculated in lines 6-9. Otherwise, more data centers must beutilized. Keep doing so until all user requests are satisfied andthe initial SAC is attained. For the next step, we iterativelyupdate SAC by deactivating servers from data centers withhigh electricity price to minimize the total electricity cost.First, with initial SAC derived aforementioned, an initialcomputation capacity hn and electricity cost EC can becalculated by solving problem (11)-(15). Then we reduce thenumber of activated servers from the most expensive datacenter with a bisection method as shown in lines 14-26. Foreach new SAC
′, its corresponding minimum electricity cost
EC′
is achieved. If there is a comparatively lower electricitycost, we set it as new EC and seek to deactivate moreservers in this data center as indicated in lines 19-21. Oncethe new cost EC exceeds the temporarily optimal cost, nomore severs shall be deactivated thus we should recover some
6 VOLUME *, *
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
overly deactivated servers in lines 22-24. Finally, with algo-rithm converging, the optimal SAC setting and the minimumelectricity cost EC is attained and the problem (11)-(15) issolved.
Define the maximum server number in each data center asMS, then the complexity of our algorithms is O(NlogMS)which mainly arises in Case II (Algorithm 3), where weiteratively deactivate data centers via a bisection method inlines 14-26. As to a data center n with San activated servers inthe initial SAC, log2San iterations are implemented. When itcomes to the worst case, all data centers are checked, leadingto
∑Nn=1 log2S
an = O(NlogMS) iterations in all. Most
importantly, problem (11)-(15) is a convex problem givena SAC, therefore we can get a global optimal solution inpolynomial time.
V. EVALUATIONS AND ANALYSESIn this section, we perform our evaluations in MATLAB onan Intel workstation with 4 processors clocking at 3.3 GHZand 8 GB of RAM. An optimization toolbox fmincon isutilized to solve (11)-(15) given a SAC.
A. EVALUATION SETTINGSEvaluation parameters involved are shown in Table 1. As aserver’s power level is usually hundreds of Watts, the scalingfactor r and coefficient B can be calculated accordingly.
TABLE 1: Parameters involved in evaluations
Parameter Value
Number of front portals 4− 8Number of user requests 80− 160000Number of data centers 3− 24Number of servers (3− 9) (×104)Server delay (s) [17] 0.01Server operating frequency set (GHz) [0.8, 1.0, 1.2, 1.4, 1.6, 1.8,
2.0, 2.2, 2.4, 2.6, 2.8]Scaling factor r 6× 10−10
Coefficient B 1.8× 10−26
Proportion of static power consumption 0.1− 0.6Size of user requests (MI) 4000− 10000Total request rate (units/s) (0.7− 1.3) (×105)
Unless stated otherwise, the other basic parameters are setas follows:
(1) Data center parameters: Three data centers located inBrussels, Flanders, Limburg and four front portals inBelgium are tested. Server number of these three datacenters are 15000, 30000 and 10000, respectively.
(2) REG parameters: We assume there are solar panels in-stalled in geo-distributed data centers. All REG tracesare based on the real-world data [28] of the real-timeestimations of actual solar-PV generation of Brussels,Flanders and Limburg in August, 2017.
(3) Electricity price: Electricity price traces are obtainedfrom NYiso [29]. We adopt the locational based marginalelectricity prices of central New York Control Area onAugust 15th, 2017.
(4) User request information: User requests are consideredto arrive at front portals in Poisson distribution. The aver-age request rates at four portals are 30000, 15000, 15000and 20000 units/s, respectively. Request size is uniformlyvaried between 4000 MI and 10000 MI (million instruc-tions).
B. RESULTS AND DISCUSSIONSIn this section, we conduct experiments to evaluate the im-pacts of different system parameters (static power consump-tion, server number, user requests, etc.) on electricity costand server frequencies, and assess the performance of theproposed optimal task placement strategy.
1) The Impact of Static Power ConsumptionIn some literature, the static power consumption is usuallyignored [20, 30]. However, the real fact is that static powerconsumption has a strong impact on data center managementand the electricity cost. In this test, the proportion of staticpower consumption is set from 0.1 to 0.6. We select REGat 12:00 as 1150.25, 1290.1, 1209.99 MW/h and electricityprice as 42.93, 20.27, 55.30 $/MW for the three data centers,respectively. Results concerning the impact of static powerconsumption on electricity cost corresponding to data centerswith and without solar panels are depicted in Fig. 3. It’sobserved that electricity cost rises when static power con-sumption increases, thus we should think carefully whetherit is necessary to activate more servers for more requests.Furthermore, data centers installed solar panels can saveelectricity bills significantly.
Proportion of static power consumption0.1 0.2 0.3 0.4 0.5 0.6
Ele
ctric
ity c
ost
($
/s)
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1Electricity cost with solar panelsElectricity cost without solar panels
FIGURE 3: Electricity cost with respect to different staticpower consumption
2) The Impact of REG and Electricity PriceBased on the proposed strategy, user requests are moreprone to be processed by data centers with sufficient
VOLUME *, * 7
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
REG and low electricity price. To better present the re-sults, we consider 6 data centers in this test with userrequests arriving rates as 30000, 30000, 30000, 20000 re-spectively at 4 front portals, and server numbers are15000, 30000, 10000, 20000, 5000, 25000 for 6 data centers,respectively. Unless explicitly specified, the proportion ofstatic power consumption is set as 0.2. We select electric-ity price at 12:00 as 42.93, 20.27, 55.30, 30.93, 25.27, 50.30$/MW and REG as 341.4, 1290.1, 83.5, 634.7, 322.9, 923.8MW/h for the 6 data centers, respectively. Results concerningthe impact of REG and electricity price on server activationconfiguration and electricity cost are shown in Figs. 4 and5 respectively. In Fig. 4, it is clear that more servers areactivated in data centers with large REG, thus they are en-abled to process more user requests with lower costs. In Fig.5, user requests are mostly dispatched to data centers withsmall electricity price for executing. However, as indicatedby the blue bars, lower electricity price together with heav-ier workload doesn’t definitely lead to low electricity cost,considering the variation of cost savings from the renewableenergy.
Data centersDC 1 DC 2 DC 3 DC 4 DC 5 DC 6
Num
ber
of s
erve
rs a
ctiv
ated
0
500
1000
1500
2000
2500
3000
3500
4000
Num
ber
of r
eque
sts
proc
esse
d
0
500
1000
1500
2000
2500
3000
3500
4000
Server activation configuration (RE) Number of requests processed (RE)Renewable energy generation
341.4 MW/h
1290.1 MW/h
83.5 MW/h
634.7 MW/h
322.9 MW/h
923.8 MW/h
FIGURE 4: Number of servers activated and requests pro-cessed with respect to different REG
3) The Impact of Active Server NumberIn this test, we evaluate the impact of active server numberson the electricity cost under different proportions of staticpower consumption. The active server number varies from30000 to 90000 and the proportions of static power consump-tion is set as 0.2 and 0.5 respectively. Other parameters arekept the same as the settings in V-B1. Results correspondingto electricity cost of data centers with and without solarpanels are depicted in Fig. 6. When the proportion of staticpower consumption is 0.2, it is shown that electricity costdecreases first and then increases when the active server num-ber grows. This is because when user requests are distributedto more servers, servers on each data centers process less
Data centersDC 1 DC 2 DC 3 DC 4 DC 5 DC 6
Ele
ctrici
ty c
ost
($
)
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Num
ber
of re
quest
s pro
cess
ed
0
10000
20000
30000
40000Electricity costNumber of requests processed (CE)Electricity price
5000
15000
25000
35000
42.93 $/Mh
30.93 $/Mh
55.30 $/Mh
20.27 $/Mh
25.27 $/Mh
50.30 $/Mh
FIGURE 5: Electricity cost and number of requests processedwith respect to different electricity price
requests with lower frequencies which leads to lower electric-ity cost. However, when the active server number increasesover an inflection point (80000 in this experiment), moreactivated servers consume more static power consumption,and the server frequencies are far away from the economicinterval, thus leading to the increment of electricity cost.This phenomenon can also be seen when the static powerproportion of servers is set as 0.5, where the optimal activeserver number is around 45000. The results also verify thatthere exists an optimal server activation configuration pointto process all the user requests before the deadline.
0.08with solar panels (static power proportion: 0.2)without solar panels (static power proportion: 0.2)with solar panels (static power proportion: 0.5)without solar panels (static power proportion: 0.5)
FIGURE 6: Electricity cost with respect to different activeserver numbers
8 VOLUME *, *
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
4) Impact of Total User RequestsIn this test, we evaluate the impact of total user requests onthe electricity cost. The results corresponding to data centerswith and without solar panels are shown in Fig. 7. It isobvious to observe that electricity cost increases when totaluser requests expand. In addition, the electricity cost are moresensitive to the user requests when they are at high levels.This is because the power consumption of servers increasesfaster when servers run at higher frequencies.
Total requests #1050.7 0.8 0.9 1 1.1 1.2 1.3
Ele
ctric
ity c
ost (
$/s)
0.04
0.06
0.08
0.1
0.12
0.14
0.16Electricity cost with solar panelsElectricity cost without solar panels
FIGURE 7: Electricity cost with respect to different userrequests
5) The Variation of SAC, Requests Processed andFrequency Traces in 24 HoursIn this test, we evaluate the proposed optimal task placementstrategy during 24 hours in a day adopting real REG andelectricity price traces mentioned in Section V-A. Resultsconcerning electricity cost, SAC and number of requestsprocessed during 24 hours with different REG and electricityprices at each time slot are depicted in Fig. 8. In Fig. 8 (a),the trend of electricity cost follows the trend of electricityprice except in time slots 11 − 18 when electricity price isrelatively high and renewable energy is sufficient to processuser requests. Due to the fact that REG tends to be high intime slots 7 − 21, all renewable energy is utilized to processuser requests as shown in Fig. 8 (b). Then the remaininguser requests are dispatched to time slots with low electricityprices, such as time slots 1 − 6, 7 − 15 and 20 − 24 toensure that tasks can be processed within their deadlines.In addition, when electricity price is high, the number ofservers activated by conventional energy is small in timeslots 16 − 18, while large in the remaining time slots. FromFig. 9, it’s obvious to observe that the frequency trace hassimilar trend with the electricity cost trace, from which wemay come to the conclusion that servers indeed work athigh frequencies in dealing with large user requests owingto DVFS technique we utilized. Such observation further
10000Server activation configuration (CE)Number of requests processed
(b) Number of servers activated and requests processed
FIGURE 8: Electricity cost, SAC and requests processedwith respect to different REG and electricity price in 24 hours
proves that our optimal task placement strategy based onDVFS technique can decrease the electricity cost in a multi-electricity market.
VI. CONCLUSIONSIn this paper, we focus on the electricity cost minimizationproblem in data centers powered by mixed renewable andconventional energies. An optimal task placement strategyproblem considering the spatial and temporal diversity ofREG and electricity price, as well as DVFS technique andM/G/1 queuing model is first formulated as a complicatedMINLP. Afterwards, we transform the original MINLP intoa more easier-solving form. In this strategy, user requestsare distributed to data centers with high REG first, then theremaining user requests are dispatched to those with lowelectricity price for economic operation. The optimal SACand task placement algorithm is proposed to obtain a globaloptimal solution. Numerical evaluations based on real-worldtraces have tested the impacts of different system parame-ters on electricity cost and server activation configurations.
VOLUME *, * 9
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
FIGURE 9: Server operating frequencies with respect todifferent user requests processed in 24 hours
Simulation results show that user requests arriving at frontportals are more likely to be processed in abundant REG andlow electricity price data centers and the idea of importingrenewable energy shows that data centers installed solar pan-els cost much less than those only powered by conventionalenergy. The evaluation results prove the superiority of ourtask placement strategy and provide some insights on helpingus design economic and environment-friendly data centers.
REFERENCES[1] L. Grange, G. D. Costa, and P. Stolf, “Green IT scheduling for
data center powered with renewable energy,” Future Genera-tion Computer Systems, vol. 86, pp. 99 – 120, 2018.
[2] A. Khosravi, Energy and Carbon-Efficient Resource Manage-ment in Geographically Distributed Cloud Data Centers. 2017.
[3] A. Greenberg, J. R. Hamilton, D. Maltz, and P. Patel, “Thecost of a cloud: Research problems in data center networks,”ACM Sigcomm Computer Communication Review, vol. 39,pp. 68–73, 2009.
[4] H. Xu, C. Feng, and B. Li, “Temperature aware workload man-agementin geo-distributed data centers,” IEEE Transactions onParallel and Distributed Systems, vol. 26, no. 6, pp. 1743–1753, 2015.
[5] J. Zhu, X. Li, R. Ruiz, and X. Xu, “Scheduling stochasticmulti-stage jobs to elastic hybrid cloud resources,” IEEETransactions on Parallel and Distributed Systems, vol. 29,no. 6, pp. 1401–1415, 2018.
[6] X. Zhao, T. Peng, X. Qin, Q. Hu, L. Ding, and Z. Fang,“Feedback control scheduling in energy-efficient and thermal-aware data centers,” IEEE Transactions on systems man andcybernetics, vol. 46, no. 1, pp. 48–60, 2016.
[7] M. Zotkiewicz, M. Guzek, D. Kliazovich, and P. Bouvry,“Minimum dependencies energy-efficient scheduling in datacenters,” IEEE Transactions on Parallel and Distributed Sys-tems, vol. 27, no. 12, pp. 3561–3574, 2016.
[8] Y. Wang, X. Lin, and M. Pedram, “A stackelberg game-basedoptimization framework of the smart grid with distributed pvpower generations and data centers,” IEEE Transactions on
Energy Conversion, vol. 29, no. 4, pp. 978–987, 2014.[9] H. Yuan, J. Bi, W. Tan, M. Zhou, B. H. Li, and J. Li, “Ttsa:
An effective scheduling approach for delay bounded tasksin hybrid clouds,” IEEE Transactions on Systems, Man, andCybernetics, vol. 47, no. 11, pp. 3658–3668, 2017.
[10] L. Yu, T. Jiang, Y. Cao, and Q. Qi, “Joint workload and batteryscheduling with heterogeneous service delay guaranteesfordata center energy cost minimization,” IEEE Transactions onParallel and Distributed Systems, vol. 26, no. 7, pp. 1937–1947, 2015.
[11] H. Ninad, P. Sudeep, S. H J, A. A. Maciejewski, M. A. Oxley,and J. Eric, “Minimizing energy costs for geographicallydistributed heterogeneous data centers,” IEEE Transactions onSustainable Computing, pp. 1–14, future issue.
[12] M. Ghamkhari and H. Mohsenianrad, “Energy and perfor-mance management of green data centers: A profit maximiza-tion approach,” IEEE Transactions on Smart Grid, vol. 4, no. 2,pp. 1017–1025, 2013.
[13] T. Chen, Y. Zhang, X. Wang, and G. B. Giannakis, “Robustworkload and energy management for sustainable data cen-ters,” IEEE Journal on Selected Areas in Communications,vol. 34, no. 3, pp. 651–664, 2016.
[14] S. Chen, S. Irving, and L. Peng, “Operational cost opti-mization for cloud computing data centers using renewableenergy,” IEEE Systems Journal, vol. 10, no. 4, pp. 1447–1458,2016.
[15] A. N. Toosi, C. Qu, M. D. de Assunção, and R. Buyya,“Renewable-aware geographical load balancing of web appli-cations for sustainable data centers,” Journal of Network andComputer Applications, vol. 83, pp. 155–168, 2017.
[16] T. Chen, X. Wang, and G. B. Giannakis, “Cooling-awareenergy and workload management in data centers via stochas-tic optimization,” IEEE Journal of Selected Topics in SignalProcessing, vol. 10, no. 2, pp. 402–415, 2016.
[17] L. Gu, D. Zeng, A. Barnawi, S. Guo, and I. Stojmenovic, “Op-timal task placement with qos constraints in geo-distributeddata centers using dvfs,” IEEE Transactions on Computers,vol. 64, no. 7, pp. 2049–2059, 2015.
[18] S. Wang, Z. Qian, J. Yuan, and I. You, “A dvfs based energy-efficient tasks scheduling in a data center,” IEEE Access,vol. 5, pp. 13090–13102, 2017.
[19] Z. Tang, L. Qi, Z. Cheng, K. Li, S. U. Khan, and K. Li, “Anenergy-efficient task scheduling algorithm in dvfs-enabledcloud environment,” grid computing, vol. 14, no. 1, pp. 55–74, 2016.
[20] H. Lei, R. Wang, T. Zhang, Y. Liu, and Y. Zha, “Amulti-objective co-evolutionary algorithm for energy-efficientscheduling on a green data center,” Computers & OperationsResearch, vol. 75, pp. 103–117, 2016.
[21] C. Wu, R. Chang, and H. Chan, “A green energy-efficientscheduling algorithm using the dvfs technique for clouddatacenters,” Future Generation Computer Systems, vol. 37,pp. 141–147, 2014.
[22] A. Rahman, X. Liu, and F. Kong, “A survey on geographicload balancing based data center power management in thesmart grid environment,” IEEE Communications Surveys Tu-torials, vol. 16, no. 1, pp. 214–233, 2014.
[23] M. Dayarathna, Y. Wen, and R. Fan, “Data center energyconsumption modeling: A survey,” IEEE CommunicationsSurveys Tutorials, vol. 18, pp. 732–794, Firstquarter 2016.
[24] S. Bhandari, S. K. Sharma, and X. Wang, “Latency minimiza-tion in wireless iot using prioritized channel access and dataaggregation,” pp. 1–6, 2017.
[25] W. Wang, Y. Sun, K. Salamatian, and Z. Li, “Adaptive path iso-lation for elephant and mice flows by exploiting path diversityin datacenters,” IEEE Transactions on Network and ServiceManagement, vol. 13, no. 1, pp. 5–18, 2016.
10 VOLUME *, *
Wang et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
[26] M. Ahmed, I. Ahmad, and D. Habibi, “Load-adaptive resourcemanagement for green wireless-optical broadband access net-work (woban),” Journal of Lightwave Technology, vol. 34,no. 10, pp. 2359–2370, 2016.
[27] G. Giambene, M/G/1 Queuing Theory and Applications,pp. 367–413. Boston, MA: Springer US, 2014.
[30] Q. Wang, Y. Kanemasa, J. Li, C. An Lai, M. Matsubara, andC. Pu, “Impact of DVFS on n-tier application performance,”Conference on Timely Results in Operating Systems, p. 5,2013.
DR. RAN WANG (M’17) is currently an assistantprofessor at College of Computer Science andTechnology, Nanjing University of Aeronauticsand Astronautics (NUAA), and Collaborative In-novation Center of Novel Software Technologyand Industrialization, Nanjing, P.R. China. He re-ceived his B.E. in Electronic and Information En-gineering from Honors School, Harbin Institute ofTechnology (HIT), P.R. China in July 2011 andPh.D. in Computer Science and Engineering from
Nanyang Technological University (NTU), Singapore in April 2016. He wasa research fellow with the School of Electrical and Electronic Engineering,Nanyang Technological University (NTU), Singapore from October 2015to August 2016. He has authored or coauthored over 30 papers in top-tierjournals and conferences. He received the Nanyang Engineering DoctoralScholarship (NEDS) Award, Singapore and innovative and entrepreneurialPh.D. Award of Jiangsu Province, China in 2011 and 2017, respectively.His current research interests include intelligent management and controlin smart grid, network performance analysis and evolution of complexnetworks, etc.
MS. YIWEN LU is currently pursuing her Mas-ter Degree in Software Engineering at College ofComputer Science and Technology, Nanjing Uni-versity of Aeronautics and Astronautics (NUAA),P.R. China. She received her B.E. Degree innetwork engineering from Qingdao University inJune 2016, during which she received various na-tional scholarships. Her current research interestincludes intelligent management and control insmart grid and data centers.
DR. KUN ZHU (M’15) is currently a Professorin the College of Computer Science and Tech-nology, Nanjing University of Aeronautics andAstronautics, China. He received his Ph.D. degreein 2012 from School of Computer Engineering,Nanyang Technological University, Singapore. Hewas a research fellow with the Wireless Commu-nications, Networks, and Services Research Groupin University of Manitoba, Canada. His researchinterests include resource allocation in 5G, wire-
less virtualization, and self-organizing networks. He has served as TPC forseveral conferences and reviewer for several journals.
DR. JIE HAO (M’16) received her BS degreefrom Beijing University of Posts and Telecommu-nications, China, in 2007, and the Ph.D. degreefrom University of Chinese Academy of Sciences,China, in 2014. From 2014 to 2015, she hasworked as post-doctoral research fellow in theSchool of Computer Engineering, Nanyang Tech-nological University, Singapore. She is currentlyan Assistant Professor at College of ComputerScience and Technology, Nanjing University of
Aeronautics and Astronautics, China. Her research interests are wirelesssensing, visible light communication and etc.
DR. PING WANG (M’08, SM’15) received thePh.D. degree in electrical engineering from theUniversity of Waterloo, Canada, in 2008. She iscurrently an Associate Professor with the Depart-ment of Electrical Engineering and Computer Sci-ence, York University, Canada. Before that, shewas with Nanyang Technological University, Sin-gapore. Her current research interests include re-source allocation in multimedia wireless networks,cloud computing, and smart grid. She was a co-
recipient of the Best Paper Award from the IEEE International Conferenceon Communications in 2007 and the IEEE Wireless Communications andNetworking Conference in 2012. She has been serving as an AssociateEditor for several journals including the IEEE Transactions on WirelessCommunications, the EURASIP Journal on Wireless Communications andNetworking, and the International Journal of Ultra Wideband Communica-tions and Systems.
DR. YUE CAO (M’16) received the Ph.D. degreefrom the Institute for Communication Systems(ICS), 5G Innovation Centre (5GIC), at Univer-sity of Surrey, Guildford, UK in 2013. He wasa Research Fellow at the ICS until September2016, and Lecturer in Department of Computerand Information Sciences, at Northumbria Univer-sity, Newcastle upon Tyne, UK until July 2017,and currently the Senior Lecturer since August2017. His research interests focus on Intelligent
Mobility. He is the Associate Editor of IEEE Access and InternationalJournal of Vehicular Telematics and Infotainment Systems.
![Page 1: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/1.jpg)
![Page 2: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/2.jpg)
![Page 3: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/3.jpg)
![Page 4: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/4.jpg)
![Page 5: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/5.jpg)
![Page 6: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/6.jpg)
![Page 7: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/7.jpg)
![Page 8: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/8.jpg)
![Page 9: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/9.jpg)
![Page 10: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/10.jpg)
![Page 11: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/11.jpg)
![Page 12: Northumbria Research Linknrl.northumbria.ac.uk/36340/1/Wang et al - An... · cost minimization problem is modeled as a mixed integer linear program (MILP). L. Yu et al. [10] optimize](https://reader034.documents.pub/reader034/viewer/2022052104/603eb43943f0e852c51774a6/html5/thumbnails/12.jpg)
