HeporCloud: An energy, performance and cost e cient ... · run users’ applications in containers,...

HeporCloud: An energy, performance and cost efficientresource orchestrator for hybrid heterogeneous cloud

computing environments

Ayaz Ali Khana, Muhammad Zakaryaa,∗, Izaz Ur Rahmana, Rahim Khana, Rajkumar Buyyab

aDepartment of Computer Science, Abdul Wali Khan University, Mardan, PakistanbSchool of Computing and Information Systems, University of Melbourne, Australia{ayazali,mohd.zakarya,izaz,rahimkhan}@awkum.edu.pk,[email protected]

Abstract. In major IT companies such as Google, Rackspace and Amazon AWS, virtual-isation and containerization technologies are usually used to execute customers’ workloadsand applications – as part of their cloud computing services offering. The computationalresources are provided through large-scale datacenters, which consume substantial amountof energy and, consequently, affect our environment with global warming. Since long, Googlerun users’ applications in containers, Rackspace offer bare-metal hardware, whereas AWSrun them either in VMs (EC2), containers (ECS) and/or containers inside VMs (Lambda);therefore, ensuring efficient resource management a tedious activity. The role of a resourcemanagement system is of the greatest importance, principally, if IT companies practice var-ious kinds of sand-boxing technologies, such as bare-metal, VM and/or containers, in theirdatacenters (hybrid platforms) to offer quality services to customers. The absence of a sin-gle, workload-aware resource manager creates questions on datacenters energy efficiency andperformance of the workloads. In most public clouds, resources are idle as peak demandfor resources is not regular; therefore, wasting energy, resources and costs. Energy could besaved and performance can be maintained to the expected level through efficient resourcemanagement tools. In this paper, we demonstrate the possibility of: (i) using workload-awareresource managers in hybrid clouds; and (ii) achieving energy and cost savings, in heteroge-neous hybrid datacenters such that the workload performance is not affected, negatively. Ourempirical evaluation suggests that single scheduler is approximately 7.62% more energy andperformance efficient than distributed schedulers. Moreover, our proposed resource managercan save approximately 30.47% energy at cost of only 2.14% loss in workload performance.

Keywords: datacenters, virtualisation, containerization, resource management, server con-solidation, workload migration, energy efficiency, performance

1 Introduction

National energy supply, water complications, rising fuel costs, global warming, and computationalbusiness economics altogether bring the need for energy and performance, consequently, cost effi-cient computing into sharp focus. Reduction in coal-based power plants, particularly, in the UK,offering a projected energy safety margin [capacity and demand ratio] of only 0.29% in 2017 [1],and the closure of nuclear power plants in France and Germany, carry the very real danger of poweroutages and load shedding in near future. Due to increase in renewables, a slight increase in theUK energy safety margin can be seen in 2018 (uptake from 29% to 36%). If we assume compara-ble consumption rates to the world of approximately 3.0% of overall energy consumption, a 9.6%growth in datacenter energy efficiency might represent two times increase in such an energy safetymargin [1], [2]. Furthermore, [1] also suggests that, due to migration from internal systems to thecloud, datacenter energy consumption will remain constant until 2020. However, due to increase inmobile users and services, Internet of Things (IoT), and computing at scale, an increasing trend inenergy consumption of the current datacenters can still be seen. Such an increase in energy con-sumption and the expected level of service performance would certainly affect the environmentalsustainability (3% Greenhouse gases), user’s monetary costs and cloud economics [e183.98billions

2 Khan A.A., Zakarya M., Rahman, I.U., and Khan R., Buyya R.

in 2016 to e217.05billions in 2017 - 18% increase]. For example, Amazon AWS experienced approx-imately 1% reduction in their sales due to only 100 milliseconds loss in performance. Therefore, itis essential to look deeply into the problem and identify possible causes, opportunities and appro-priate solutions for energy savings and performance improvements (as agreed in SLA document -service level agreement), therefore, cost savings [2], [3].Energy supply problems, water complications and rising fuel costs suggest the need to examine forthe causes of rising energy consumption in datacenters and seeks to eliminate the causes and/ormanage them through possible solutions under workload performance constraints. The increasingnumber and use of Information & Communication Technology (ICT) equipment in cloud datacen-ters has a consequential impact on energy consumption and workload performance levels. Likewise,the decreasing use of non-renewable energy sources, such as coal, increases the need to develop so-lutions for managing datacenters resources in order to minimize the growing levels of energy use,worldwide [2]. In respect of the former statement, datacenter’s resources are usually under-utilised;and idle thus making it possible to use methods like virtualisation and containerisation to saveenergy. In respect of the later statement, workloads might be moved, across resources powered byvarious energy production methods such as coal and renewables, when it is essential (as renewablesare intermittent) or more beneficial (cost efficient) to do so.Virtualisation allows same resources to be shared among different users (applications), that in-creases resource utilization and hence energy savings through consolidation. Along with thesebenefits, virtualisation and consolidation could also produce performance related issues due to mi-gration and co-location (when similar workloads compete for same resources) that could lead toincreased user monetary costs, Virtual Machines (VMs) runtime and hence less energy efficiency.Furthermore, besides decreasing energy consumption using consolidation supported by virtualisa-tion, cloud computing can also be used to gain energy efficiency through efficient resource man-agement and scheduling [4].Containers have almost replaced Virtual Machines (VMs) as compute instance of choice in large-scale cloud platforms such as Google and Amazon AWS. Compared to VMs, containers have lowerdeployment overhead and can yield the best performance. Various applications have different busi-ness goals; some of them would run efficiently inside VMs while some might perform best on baremetal or inside containers. Furthermore, by mixing containers with traditional VMs (run contain-ers inside VMs), maximum utilization of the resources is assured through consolidation. However,this might create workload performance issues when containers and VMs are being migrated acrossheterogeneous resources. Moreover, if workloads are running over various platforms in a particu-lar datacenter, then there would be various migratable entities such as containers, VMs, hybrid(containers|VMs) and bare-metal workloads. Some of them would be more effective than the othersand vice versa. For example, inter-platform migrations may occur in a particular platform; andintra-platform migrations may occur within platforms.The goal of the present research is to explore further savings as may be possible through approachessuch as efficient scheduling and consolidation (resource management) to decrease datacenters en-ergy consumption in such a way that the workload (VMs) performance (runtime) is not negativelyaffected due to resource heterogeneity. The objective of this research is to pact with the above chal-lenge by advising a reference architecture and a single, platform-independent, resource manager.The challenge should be to determine the right abstractions to enable the design of an integratedservice leveraging the core mechanism – without the implementation of dedicated services, such asindividual scheduler and platform-specific monitoring, for each kind of sandboxes. We, then, pro-pose a centralised, workload aware scheduler and a consolidation (orchestration) technique whichreduces the datacenter’s energy consumption, therefore, costs and increases workload performance.We perform extensive simulations of the proposed architecture, scheduler, and orchestrator, usingreal cloud workloads (datasets) from major service providers such as Intel [5], Microsoft Azure [6]and Google [7] clusters that correspond to HPC (bare-metal), virtualisation and containerisationworkloads, respectively. The major contributions of the research conducted in this paper are:

– a reference architecture and a single, platform-independent, resource manager is advised;

Title Suppressed Due to Excessive Length 3

– an energy, performance and cost efficient resource scheduler is presented that is able to managecloud infrastructures running different kinds of sand-boxing technologies;

– an orchestrator is suggested that decides energy, performance and, therefore, cost efficientworkload migrations;

– a cloud simulator is developed in order to evaluate various sand-boxing technologies, at thesame time; and

– investigate the impact of datacenter resource configuration (physical order of hosts) on energyconsumption and workload performance.

The rest of the paper is organized as follows. In Sec. 2, we explain datacenters energy consumption.In Sec. 3, we discuss the scheduling and consolidation problem. In Sec. 4, we propose HeporCloud –a hybrid heterogeneity aware technique that decide efficient workload placement and migration. Sec.5 describes various models to demonstrate energy and performance heterogeneities of various cloudplatforms. We validate HeporCloud using real workload traces from Google and Azure clusters inSec. 6 and show that HeporCloud is more energy, performance and, therefore, cost efficient thanthe existing techniques. We offer an overview of the related work in Sec. 7. Finally, Sec. 8 concludesthe paper and describes future research directions.

2 Background

Various studies [1], [8] suggest that worldwide datacenters are, largely, not highly utilised; andsignificant energy could be saved through virtualisation, containerisation and mix of virtualisation,containerisation (virtualised containers) technologies. These various sand-boxing technologies offerbetter opportunities for resource scheduling and consolidation. For example, workload could bescheduled on resources where its performance may be optimal or near to optimal. Furthermore,if certain workloads perform the worst on particular resources, or the resource utilisation is low;they could be migrated to other resources to increase performance or energy efficiency. However,consolidation involves migrations that can be expensive in terms of performance loss and energyconsumption [8]. Consolidation of VMs and containers have been studied, largely, in the cloud lit-erature [4], [8], [9] in order to decrease datacenter’s operational, energy costs and improve serviceperformance. Moreover, workload consolidation might be helpful in scenarios where datacentersresources are operating using various energy sources such as coal and renewables – as renewablesare more cheaper than coal, however, intermittent. Therefore, this might be essential to moveworkloads from renewables to energy grid when they are not available; and to renewables fromenergy grid when it is more cost effective [10].Moreover, cloud workloads vary in their resource consumption and performance requirements thatcould make resource scheduling and consolidation a tedious activity [11], [12]. For example, migra-tion of a VM and container could be worthless as the effort should be wasted if the VM or containerterminate during or just after the migration is completed [4]. Moreover, similar workloads may per-form quite differently on various platforms; some of them may execute faster on containers thanVMs, some will perform worse on containers than bare-metal and vice versa. Similarly, certaincloud users may need full access to bare-metal resources in order to get total control on theirprovisioned hardware. This is also evidenced through the recent introduction of AWS bare-metalinstances; that will, probably, soon force IaaS (Infrastructure as a Service) providers to rethink ofusing various platforms in their datacenters. Therefore, complexities would arise when the cloudprovider uses a mix of these sand-boxing technologies – in order to maximise their resource usageand reduce their operational costs. In such scenario, certain workloads may perform worst on VMs,but, would execute faster over the containers, virtualised containers or bare-metal (non-virtualised)platforms [11]. Lower execution times may mean higher energy efficiency, lesser users costs andvice versa; however, energy efficiency may also relates to the sand-boxing technologies, workloadtypes and hardware energy consumption profiles [8].This creates opportunities for hybrid clouds that implement various sand-boxing technologies,


such as the Intel’s CIAO (Cloud Integrated Advances Orchestrator)1, Magnum2, Kolla3 and,later on, appropriate workload placement and migration decisions are made. The goal could beachieved through clustering the available resources such that each cluster corresponds to a par-ticular sand-boxing technology. Furthermore, each cluster may have either its own scheduler orshare a centralised scheduler. Using individual schedulers for each sand-boxing technology such ascontainerization, virtualization, virtualised containers, bare-metal may not be appropriate in termsof energy efficiency and performance; due to the lack of entire datacenter state and resource usageinformation at each scheduling (platform) level. If these schedulers, that belong to various plat-forms and sand-boxing technologies, can communicate and share entire datacenter state with eachother (i.e. centralised scheduler); appropriate energy and performance efficient workload placementand migration decisions could be taken [2]. Moreover, this will provide support for inter-platformand intra-platforms migrations; which are, to the best of our knowledge, unexplored in the existingliterature of cloud computing.

3 Problem Description

VMs (and recently containers) have enabled the fast adoption of the cloud computing environ-ment, and the needs, in terms of utility computing, progressed to assimilate numerous kinds ofsand-boxing technologies such as virtualisation, containerization, bare-metal and virtualised con-tainerisation [13]. Generally, HPC users would prefer accessing raw hardware (bare-metal) to deploytheir applications which reduces the risks of performance loss due to virtualisation. This is evi-denced through the recent introduction of the bare-metal instances in the AWS cloud; which allowsusers to have full control over their provisioned resources. Moreover, certain workloads would per-form better on containers than VMs and vice versa. For example, bank applications would runmore securely in VMs than containers. In such circumstances, as shown in Fig. 1, variations inapplications runtimes and, therefore, user monetary costs would create questions on datacenterenergy consumption, workload performance and cloud economics i.e. users monetary costs anddatacenters energy bills.Big data actors such as Intel, Google and Amazon AWS investigate cutting-edge solutions builton the idea of VMs and/or containers. The needs of these developments have a key impact onIaaS management systems which are used to develop dedicated services to cope with resource andworkload heterogeneities. Along with the complexity of resource management system, till now,these evolutions have been accomplished independently, without demonstrating whether accurateabstractions could allow the management of any kind of sand-boxing technologies in an integratedway (centralised scheduler) or in a distributed style (distributed schedulers). Furthermore, howvarious combinations of resource allocation and migration policies would affect datacenter energyconsumption and workload performance.The aim of this research is to: (a) explore and develop a management system architecture to pro-vision and operate several kinds of sand-boxing systems on top of IaaS clouds [14]; (b) investigatehow various combinations of resource allocation and consolidation policies will affect energy andperformance efficiencies of the datacenter resources; (c) propose workload aware resource allocationand consolidation techniques to reduce energy consumption, therefore, costs and increase work-load performance; and (iv) investigate the impact of inter-platform and intra-platforms migrationson infrastructure energy efficiency and workload performance, therefore, costs. In public clouds,reasonable best efforts would mean no loss in performance, as performance loss would certainlyaffect the SLA’s; and violation to SLA’s would require a penalty of service providers. Whereas, inprivate clouds, increase in performance would be essential for certain kinds of workloads such asHPC, database operations and etc.

1 https://ciao-project.github.io/

2 https://wiki.openstack.org/wiki/Magnum

3 http://docs.openstack.org/developer/kolla/


Fig. 1: Variations in applications performance when running over various sand-boxing technologies[from left to right and top to bottom: VMs, containers, containers|VMs and bare-metal]

3.1 Mathematical Formulation

Considering the above diverse requirements and circumstances (i.e. increased runtime, decreasedperformance and increased energy consumption), our aim is to develop a consolidation modelwhich: (i) predicts the energy and performance of the workload; (ii) derive a correlation among thepredicted quantities (containers and VMs) to decide migrations; and (iii) finally migrate the bestmigratable entity to achieve better results in terms of energy consumption and workload perfor-mance. The proposed approach is an attempt to minimize datacenter energy consumption (w.r.tservice providers) without reducing the workload performance, even if migrated (w.r.t users). Wecan express the workload migration as a multi-objective optimization problem which comprisesthree nominal cost types namely energy consumption cost (Ecc), monetary cost (Mc), and work-load performance cost (Wpc). Three entities (i.e., service providers, workloads, and consumers)are involved in the overall process and based on their characteristics each is mapped to a uniqueobjective criterion as described below;

– Service providers → minimize the amount of energy consumed – Ecc,

– Workloads → achieve their desirable performance level at the agreed costs (to meet SLAs) –in terms of runtime (R), with performance being defined as the inverse of R and the objectivehere, is to minimize or maintain R – Wpc, and

– Users → are billed accordingly i.e. minimize cost or maintain the agreed cost – Mc.

It can be intuitively understood that Mc is proportional to R (users are billed according to theirworkload runtimes), and therefore, if objective (ii) is satisfied then objective (iii) is also automat-ically satisfied; hence objective (iii) is ignored in our current work. Consequently, the objectivesof our bi-objective optimization problem become to minimize both energy (E) and runtime (R).Mathematically, this can be expressed as Eq. 1:


f =

min(E) whereE =

∑platformsi=1 Pi and P =

∑hostsj=1 Ej

max (Performance) ⇐ min(R) where R =∑Pj=1Runtimej

min(R) ⇐min(C)

(1)

where P denotes the amount of energy consumed in a particular platform. The constraints are: (i)each container or VM is mapped to only one host at a time; and (ii) the number of containers|VMson a VM|host cannot exceed the VM|host capacity [15].Gupta et al. [16] suggested the ERP (Energy Response time Product) metric to capture the trade-off among energy, performance and, therefore, cost; which is widely accepted as a suitable metricto capture similar trade-offs [17]. Minimizing ERP can be seen as maximizing the “performance-per-watt” – with performance being defined as the inverse of mean response time. In our case,performance is determined through R that can be assumed similar to response time (based on thetime factor). Therefore, we revise the given name of this metric to the Energy Runtime Product(ERP ). The ERP is given by Eq. 2:

ERP = E ×R (2)

We assume E and R both of comparable magnitudes. However, in more complex scenarios, if onedominates the other, then ERP can be expressed as ERP = α.E × β.R (where α and β are thedomination factors for E and R, respectively) [16], [17]. Theoretically, the objective of the abovebi-objective optimization problem is to minimize and evaluate the behaviour of ERP for differentscheduling and consolidation approaches with migration techniques, given by Eq. 3:

min(ERP ) (3)

From an experimental point of view, ERP of every host is estimated using energy and runtimeprediction techniques, as discussed in Sec. 4. The incoming workload is assigned and/or migratedto the host having the least ERP . We investigate through plausible assumptions and event drivensimulations in CloudSim [18], how various scheduling and consolidation with migration policies ina heterogeneous cloud platform, would affect the energy consumption, workload performance, andcost when various kinds of workloads are taken into account.

4 HeporCloud - System Architecture and Resource ManagementAlgorithms

In this section, we propose “HeporCloud” that uses a single scheduler and orchestrator to manageheterogeneous hybrid datacenters resources, energy and performance, therefore, cost efficiently.The HeporCloud architecture is shown in Fig. 2.

4.1 The HeporCloud framework

We propose an architecture/resource manager “HeporCloud” that enables the management ofdifferent sand-boxing solutions; and evaluate the performance of the proposed approach using sim-ulations and real cloud test-bed. The resource manager, as shown in Fig. 3, consists of three majormodules: (i) a single scheduler [HeporCloudScheduler]; (ii) an orchestrator [HeporCloudOrchestra-tor]; and (iii) HeporCloudStat which is responsible to collect node level statistics such as utilisationlevel, workload runtimes and etc. The HeporCloudScheduler and HeporCloudOrchestrator are in-stalled on a separate host while HeporCloudStat is installed on every host.The HeporCloudScheduler and HeporCloudOrchestrator are aware of the whole infrastructure(multiple platforms); and use predictors for workload-aware resource allocation and migrationdecisions.


VMs

Containers

Containers/VMs

Bare‐Metal

D A T A C E N T E R Reconfiguration

Orchestrator

Predictor

MigrationStatistics

Scheduler

Predictor

WorkloadRepository

Demand

Placement

Failed Schedule

Fig. 2: The proposed HeporCloud architecture for hybrid clouds

Hybrid Cloud Platform

App App App

Container Container

HeporCloudStat

Hardware

App App App

VM VM

HeporCloudStat

Hardware

App App App

Container Container Container

VM VM

HeporCloudStat

Hardware

App

HeporCloudStat

Hardware

HeporCloudOrchestrator

HeporCloudScheduler

HeporCloud Networking

HeporCloud User Interface & CLI

Fig. 3: The proposed HeporCloud framework [from an implementation point of view]

The HeporCloud scheduler: From an implementation point of view, the HeporCloudSchedulercan be assumed as a centralised scheduler that interconnects various hardware technologies suchas virtualised, containerized, virtual containerized (i.e. containers run inside VMs) and bare-metal,in the cloud platform. In order to optimise resource allocation and management, the schedulermaintains a history of resource utilisation, various applications, their energy consumption and per-formance.The scheduler predicts the future workload type and categorize it according to its energy con-


Algorithm 1: The HeporCloud scheduler

Input: workload wi, history HOutput: schedule wi

1 predict wi from the history H;2 categorize wi from the history H with respect to energy and performance;3 if wi ∈ { HPC, VM, Container|VM, Container } then4 p ← select the best platform to run wi;5 for each host ∈ p do6 select the best host h ∈ p;7 if h can accomodate wi then8 schedule wi accordingly;9 end if

10 end for

11 end if12 return a

User

Job

Platforms

Jobs details

Feature history

Estimator

Energy

X

Performance

Platform

Prediction core

Hosts details

Performance

Fig. 4: Prediction of workload platform – in the first phase, similar jobs are being collected fromthe feature history; in the second phase, a best platform is being chosen [19]

sumption and performance. The prediction module, shown in Fig. 4, looks for a platform wherethe application’s performance is best at minimum energy cost. In the first phase, the runtime forthe submitted job is predicted. Along with runtime, other job characteristics such as its submit-ting user, job name, logical job name and resource (cpu, memory) requirements are also taken intoaccount. In the second phase, the prediction module searches the features history of previous exe-cuted jobs on various platforms (hosts) and their performance or runtimes. The idea of these twophases stems, basically, from [19], [20]. In the third phase, the estimator chooses the best platform(host) that could run the workload with the minimum product of energy consumption and run-time. Thus, if the predicted workload falls within the category of HPC (bare-metal applications),VM, container|VM, or container, then the scheduler runs it, as appropriate, either on the mostenergy-performance efficient bare-metal, VM, virtualised container, or container, respectively. Notethat the predictor is a history-based, which updates the previous history, at a separate host (NAS- network area storage), at regular intervals of time.

The HeporCloud orchestrator: The HeporCloudorchestrator runs periodically or when thereare consolidation opportunities due to lower resource demand. From implementation point of view,


when the utilisation level of hosts either increases or decreases from some pre-defined thresholdvalues (Ulow and Uup), the orchestrator is activated to optimise the datacenter’s state throughconsolidation with migration technique. The orchestrator predicts whether a workload (runningon bare-metal), VM, container or nested container (containers running inside VMs) should be mi-grated in order to minimize energy consumption and performance loss. The steps involved in theoptimisation are described in Alg. 2. In the first phase, all migratable entities (workload, VM, con-tainer) are searched for. In the second phase, a suitable platform is estimated as its target platformusing a predictor and the features history. This could be achieved using the HeporCloutScheduler,as shown in Fig. 4. Moreover, various characteristics of the migratable entities on both source andtarget hosts, such as energy consumption, remaining runtime (execution time) and performancerequirements, are considered during the migration decisions, as shown in Alg. 3.

Algorithm 2: The HeporCloud orchestrator

Input: datacenter state siOutput: migration decision mdi

1 migration map mp← optimize(si);2 if mp consists of workload, VM, container, function then3 while mp.isEmpty 6= TRUE do4 wc← predict the best candidate for migration using Alg. 3 and Alg. 4;5 scheduler.inform(wc);6 mp.wc← null;

7 end while

8 end if9 return b

Furthermore, if there are multiple migration opportunities, then priority is given to that migratableentity that could (a) spend less energy on migration; and (b) save more energy or money (goodperformance - less runtime) after the migration. The savings are calculated using Alg. 4. In orderto calculate the remaining runtime of a migratable entity rtime on the target platform, its totalruntime is predicted with the predictRuntime() routine, using the features’ history, as shown inFig. 5. There are already many literature focused on application runtimes prediction [20], [21].Prediction of an application runtime can be performed in two steps: (i) find similar applicationsthat were executed in the past; and (ii) use a statistical approach to estimate its runtime. In respectof (i), application characteristics such as submitting user, resource requirement, past runtime canbe used to gather similar applications and their data from a database. The database is to be main-tained either on each host or over a centralized server. The accuracy of the prediction is stronglyrelated to the accuracy of the similarity measures. In respect of (ii), techniques like mean, movingaverage and regression analysis could be used to reach an estimation. Further details on runtimeprediction can be found in [20], [21].

The HeporCloudStat: The HeporCloudStat module is running on every cluster node; and isresponsible to collect node level statistic, periodically or when needed. These statistics, includingresource consumption and utilisation levels, workload runtimes (performance), submitting users,are stored on the same or on a separate host; preferably, a network area storage (NAS) server.The HeporCloudScheduler and HeporCloudOrchestrator use these details for various purposessuch as workload-aware resource allocation, host selection and prediction of effective migrations,respectively.


Algorithm 3: Feasible migration technique

Input: hybrid platform, migration map (migMap)Output: migratable entity (e)

1 platform.optimize() [call optimization module for consolidation opportunities];2 migTempMap← getMigratableWorkloads();3 migMap← NULL;4 for every host h in the hybrid platform do5 for every item i in migTempMap do6 rate← h.energy.predict(i)× h.performance.predict(i);7 [rate→ minimise the product of energy and performance];8 rtime ← i.predictRuntime()− (getCurrentT ime()− i.getSubmitT ime);9 savings← callAlgo(rate, rtime) [call Alg. 4];

10 if savings ≥ 0 then11 migMap.add(i, savings);12 remove i from migTempMap;

13 else14 do nothing;15 end if

16 end for

17 end for18 migMap.sortDesc(savings);19 e← migMap.pop();20 return e

Algorithm 4: Calculate power savings

Input: The list migratable entities L, Costmig, t, tmig, source and target hostsOutput: Return Psavings, the amount of energy that could be saved

1 for each migratable entity e ∈ L do

2 Ef ← (PCsource×µCsource

PCtarget×µCtarget);

3 if Ef > 1 then

4 4x← PCsource× µCsource

− (PCsource×µCsource

Ef);

5 if 4x > 0 then

6 toff ← currentTime (t) + tmig +[tmig×Costmig

4x

];

7 rtotal ← predictRuntime(e);8 ts = rtotal − toff ;9 Psavings = ts ×4x;

10 else11 Costmig is not recoverable;12 remove migratable entity e from L;

13 end if

14 else15 target is not more energy and performance efficient than the source;16 remove migratable entity e from L;

17 end if

18 end for19 return Psavings


Job

User

Job name

Job name

(logical)

J

User

Job name

Job name

(logical)

Feature history

Estimator Runtime

Prediction core

Fig. 5: Prediction of workload runtimes – in the first phase, similar jobs are being collected from thefeature history; in the second phase, a particular statistical method is used to estimate runtimes [20]

4.2 Resource Predictions

Application’s runtime prediction techniques have at least two major benefits: (i) efficient resourcescheduling (allocation or placement) decisions can be made – for example, VMs|containers withsimilar applications (or runtimes) can be placed on the same host [22]; (ii) if a particular hostneeds maintenance, the resource manager can estimate runtimes of all VMs|containers located onthe host – to determine when maintenance can be scheduled, and whether VMs|containers need tobe migrated or not [6]; and (iii) cost effective migration decisions can be triggered – for exampleif certain workloads perform worst (due to co-location) they can be migrated to other hosts [14].In [6], the authors have demonstrated that most runtimes of VMs in Azure cloud are relativelyshort i.e. more than 90% of runtimes are shorter. The curves show a knee around a day andthen almost flatten out; suggesting that, if a particular VM runs for a day, it will very likely runmuch longer. Moreover, the relatively small percentage of long-running VMs actually account formore than 95% of the total resource usage. These findings are very similar to Google cloud, whereapplications run in containers [7], [8].As noted in Sec. 4.1, the HeporCloud framework consists of three prediction techniques: (i) predictthe workload type; (ii) predict the workload runtime; and (iii) use workload runtimes and previousruns to predict an appropriate platform/host. In respect of (i) and (iii), appropriate resourcesand technology could be selected to run these workloads. In respect of (ii), appropriate migrationdecisions could be made. Cortez et al. [6] suggests that same subscriptions have almost similarworkloads, largely, with similar CPU utilisations. The author have used Fast Fourier Transform(FFT) to find periodicity in various workloads and categorized them as either potentially interactiveor delay-insensitive (batch workloads). Moreover, the literature is significantly vast that describevarious techniques, such as resource utilisation levels, priorities and submitting users, to profileand predict cloud workloads. The providers can use this knowledge in packing VMs and containerson hosts, as appropriate. For example, delay-insensitive workloads can be packed more tightly;while interactive workloads can be loosely packed onto hosts. Furthermore, the provider can avoidover-subscription of resources that run interactive workloads while allowing over-subscription forother workloads. Lastly, it is also possible to choose an appropriate sand-boxing technology (VM,container, container|VM, bare-metal) to run these workloads in an energy and performance efficientway.To simplify the implementation, we use the workload priority as a representation of its type. Thisis in-line with previous assumptions as the task’s priorities affect billing in Google cloud [23]. In ourdataset, each job is submitted along with its priority. In order to predict the workload runtimes,we use the two most important features i.e. submitting user and workload type (priority). This is


also in-line with previous work as demonstrated in [6] – users, largely, submit similar jobs in Azurecloud. We are aware that there would be other efficient ways, such as machine learning techniques(gradient boost trees, performance monitoring), to estimate runtimes; however, they might beimpractical in large environments. Once the workload type and its estimated runtimes are known,we can use the ERP of every host to select the most energy, performance and cost efficient host.The process also involves transforming the remaining runtime of a particular workload on a sourcehost to equivalent remaining runtime on a target host. This can be achieved using the z-scorenormalisation [24]. The z-score (standard score) as given by Eq. 4, can be used to calculate theprobability of a score (x) occurring within a normal distribution. Furthermore, it also provides away to compare two scores that are from different normal distributions.

z =x− µσ

(4)

Eq. 5 can be used to find runtime of the migrated workload (estimated) on the target host with givenmeans (µ, µ1) and standard deviations (σ, σ1) of source and target hosts (for normal distributions).

x− µσ

=x1 − µ1

σ1(5)

where x and x1 are the expected runtimes of the migrated workload on the source and target hosts,respectively. The left and right sides of Eq. 5 relate to the z-scores of the source and target hosts,respectively. This formulation allows us to calculate the probability of a score (expected increaseor decrease in runtime of the workload on target host) occurring within a normal distribution ofthe performance variations due to resource or platform heterogeneities. The above Eq. 5 can berewritten to find the expected runtime (x1) of the migrated workload on the target host given itsremaining runtime (x) on the source host, as Eq. 6:

x1 = σ1 ×{x− µσ

}+ µ1 (6)

For lognormal distributions, both x and x1 must be replaced with log(x), log(x1) according to thedefinitions of normal and lognormal distributions [24]. For lognormal distributions, the expectedruntime is given by Eq. 7:

x1 = exp

(σ1 ×

{log(x)− µ

σ

}+ µ1

)(7)

Note that the remaining runtime can be computed through subtracting current time from theworkload total runtime - predicted using the model as shown in Fig. 5.

5 Modelling Energy Consumption and Platforms Heterogeneities

In Sec. 5.1, we discuss how the energy consumption of a container|VM or physical host should bemeasured in simulations. In Sec. 5.2, we describe the performance of various benchmarks workloadswhen run over various platforms such as virtualisation, containerization, virtualised containers andbare-metal hardware.

5.1 Modelling Energy Consumption

We use real energy consumption data for hosts, from SPECpower4 benchmarks, to measure data-center’s total energy consumption [as described in Sec. 6.1]. However, as there is no way to measurethe energy consumption of a VM and container, directly; therefore, we can use various mathemat-ical models to get their estimated energy consumption. Another approach would be to divide the

4 https://www.spec.org/


host total energy consumption on the number of VMs or containers running on it. The total energyconsumption (P ) of a non-virtualised computer system can be estimated using the linear powermodel, given by Eq. 8; utilisation level of the CPU is directly proportional to the amount of energyconsumed [8].

P = Pidle + (Pmax − Pidle)× u (8)

where u is the CPU utilisation level, Pmax and Pidle is the energy consumption when the CPUis 100% utilised and idle, respectively. Moreover, for a virtualised/containerized host, the energyconsumption of a single VM or container can be related to the number of VMs or containersrunning on that particular host [25]. Therefore, it is reasonable to predict/estimate a VM or acontainer energy consumption with the linear CPU energy consumption model [8] using Eq. 9:

Pvm =

(Pidlen

)+Wvm × (Pmax − Pidle)× u (9)

where n is total number of VMs or containers accommodated on host, Wvm is the amount of host’resources (e.g. number of cores) allocated to the VM or container, and u is the utilisation level ofthe VM or container. The total energy consumption of a virtualised host P that runs n VMs isgiven by Eq. 10:

P =

n∑i=1

Pvmi(10)

To keep it simple, we assume the number of cores as the only resource i.e. Wvm. However, otherresources, such as CPU and memory, may also be taken in similar resource allocation decisions.The utilisation u of a VM or container refers to the real workload dataset i.e. pre-defined. The useofWvm enables us to simplify concerns by assuming each VM or container as a real host (physical);and in an infrastructure cloud, similar to VM sizes, container sizes may also be equally divided bythe number of allocated cores (hyper-threaded) out of m cores on the host, or by allocation of anamount of memory. For simplicity, we only use the number of cores (hyper-threaded):

Wvm =coresvm

m(11)

Furthermore, Pidle and Pmax are the host’s energy consumption (in Wh – Watt hours) whenit is idle and 100% utilised, respectively [25]. The above model could be used to predict theenergy consumption of a particular VM or container on a particular host at appropriate utilisationlevels [26]. If there are several containers running on a particular VM, then the total energyconsumption of the VM Pvm is given by Eq. 12:

Pvm =

m∑j=1

Pcontaineri (12)

where Pcontainer is the amount of energy consumed by a particular container or containerisedapplication; and is given by Eq. 13:

Pcontainer =

(Pvmidle

m

)+ Vc × (Pvmmax

− Pvmidle)× uc (13)

where uc is the utilisation level of the container and Vc is the fraction of VM resources (such asvCPU cores) allocated to the container. We are aware that there would be other more accuratemodels to estimate the energy consumption of a host, VM or container [27], [28], [29]; however,all these works have modelled the VM/container energy consumption as a fraction of physicalhost and its utilisation level. In the context of our work, as we use SPECpower benchmarks forhosts energy consumption that accounts for CPU, memory, disk etc.; therefore, we believe that theoverall datacenter’s energy consumption will not be affected using a different energy consumption


model. Hence, in this paper, we focus on datacenter’s energy efficiency, but, not a complete cloudenvironment, therefore, we assume that the above model is quite sufficient to measure the energyconsumption of a datacenter. For a complete cloud environment, the energy consumption of com-munication networks, and other components must be taken into account [30]. Moreover, energy dueto communication between containers|VMs located on different host must be taken into accountwhen estimating the energy consumption.

5.2 Modelling Performance

Due to non-availability of a representative cloud workload (performance-specific) and CPU per-formance benchmarks; we model resource and application heterogeneities using statistical distri-butions. First, we collect data and simulate them using monte-carlo simulations. Using mappingtechniques, we, then, relate this data to available real dataset (runtimes) in order to extract re-source and application performance parameters [24]. In this section, we discuss heterogeneities ofvarious applications when run on different platforms such as VMs, containers, virtualised containersand bare-metal hardware. Morabito et al. [31] and Felter et al. [32] have discussed virtualisation,containerization and virtualised containers. Their investigation suggests that virtualisation per-forms worse than the other techniques (∼45.76%); however, virtualised containers performs betterthan virtualisation (∼4%) and worse than containers (∼26.46%). These variations in performancecan be related to hardware or CPU platforms [33]. As described in Sec. 3, variations in runtimesaffect the customer monetary costs as well as the providers revenues. In this section, we providean overview of runtimes when various applications are being executed using various sand-boxingtechnologies on various CPU architectures.

Virtualisation: Virtualisation can increase the utilization levels of datacenter resources, however,they suffer from performance degradation due to resource contention – particularly when VMswith similar workloads compete for same resources [12]. Moreover, similar workloads (runninginside similar instance types) perform quite differently on various CPU architectures [33]. Thedistribution of runtimes of a particular workload on a specific CPU platform can be modelled aslog-normal. Moreover, certain platforms could offer the best performance for certain workloads;however, their performance is questionable for other kinds of workloads. As shown in Table 1, thesevariations in runtimes could be significant – thus have notable impact on infrastructure energyefficiency, workload performance and, therefore, users monetary costs. Note that Pxz is also acompression tool, similar to Bzip2; which is widely available in Ubunto and Fedora operatingsystems. Furthermore, we assume the Stream workload throughput (i.e. data copied in MB/s) asa proxy of VMs|containers runtimes; and the order of hosts performance is adjusted to MB/s. Aslower MB/s means longer runtime to transfer data, therefore the graph in Fig. 6, Fig. 7, Fig. 8would be interpreted with the best performance from right to left [8].

Containerisation: Containerisation is an alternative technique to virtualisation; that have beenlargely used in public datacenters such as Google. Similar to VMs, containers suffer from per-formance variations for similar workloads [34]. These variations can also be related to variousCPU platforms, as shown in Table 2. Management platforms such as Kubernetes enforce affinityconstraints (co-location) to ensure that several applications (with similarities) can be packed andplaced onto similar hosts [35]. In Kubernetes, this is accomplished by using pods groups of con-tainers. Sharma et al. [36] empirically demonstrated that containers suffer from larger performanceinterference compared to VMs. Therefore, container allocation needs to be further optimized inorder to choose the right set of neighbouring containers on a host. Kozhirbayev et al. [37] have com-pared bare-metal and two containerized platforms i.e. LXC and Docker, using various benchmarkworkloads. Their evaluation, on a real cluster, suggests that the performance of both containerizedplatforms vary for various workloads. Since, containers share the operating system, and other bi-naries of the physical host; therefore, it is possible to fit three times more containers on the hostas VMs.


Workload CPU model m1.small m1.medium(µ) (σ) Min Max CoV (µ) (σ) Min Max CoV

E5430 445.1 14.33 425.35 482.1 0.032 211.30 10.43 204.71 238.2 0.049E5-2650 470.24 13.03 443.48 518.92 0.028 223.40 3.84 217.81 233.51 0.017E5-2665 241.3 1.18 237.97 245.2 0.005 - - - - -

E5645 510.07 10.51 487.95 543.8 0.021 244.71 2.90 240.9 254.11 0.012Bzip2 E5507 620.87 28.46 578.03 715.72 0.046 312.92 14.91 295.91 332.01 0.048Pxz E5540 709.6 7.8 680.4 733.6 0.011 393.6 3.4 381.9 403.9 0.009

E5-4650 - - - - - - - - - -E5-2630 535 20 470.4 606.6 0.037 - - - - -X5560 1,680 32.5 1,625 1,755 0.019 - - - - -

———————————————————————————————————E5430 693 3 687 701 0.004 196.21 15.03 174.56 245.86 0.077

E5-2650 614 5 606 624 0.008 224.34 8.34 215.58 235.67 0.037E5645 606 7 599 628 0.012 230.83 12.19 216.93 250.03 0.053

Stream E5-2665 59.2 1.88 52.16 65.0 0.032 - - - - -Povray E5507 632 5 625 650 0.008 261.63 18.84 241.46 356.92 0.072

E5540 623.9 3.2 612.5 636.8 0.005 241.1 2.9 231.9 250.7 0.012E5-4650 - - - - - - - - - -E5-2630 128 2 120.5 134.2 0.016 - - - - -X5560 525.5 0.6 524.4 526.8 0.001 - - - - -

Table 1: Different workloads‘ runtimes over various CPU architectures and VM instances [696MBinput file to Bzip2 - Ubuntu 10.04 AMD desktop ISO file] [8], [13], [33]

Workload CPU model one vCPU two vCPUs(µ) (σ) Min Max CoV (µ) (σ) Min Max CoV

Pxz E5540 685.2 3.9 670.68 698.17 0.006 387 7.6 362.55 410.53 0.02E5-2620 19.59 1.09 15.5 23.2 0.056 - - - - -E5-2665 290.9 0.98 287.4 293.9 0.003 - - - - -E5-2680 17.95 1 14.9 20.9 0.056 8.6 1 5.6 11.9 0.116

Bzip2 E5-4650 - - - - - - - - - -i7-4600U 445.9 44.6 315.4 567.7 0.1 - - - - -

AMD 6230 14.9 0.01 14.8 15.1 0.000 - - - - -E5-2630 495 159 42.1 1,048.8 0.321 - - - - -X5560 1,622 21.75 1,580 1,667 0.013 - - - - -

———————————————————————————————————E5540 211.7 2.5 204 219.9 0.012 131.2 1.4 126.9 136 0.011E5-2620 123.47 0.43 121.64 124.73 0.003 - - - - -

Stream E5-2665 73.5 0.64 71.7 75.3 0.009 - - - - -Povray E5-2680 - - - - - - - - - -

E5-4650 - - - - - - - - - -AMD 6230 69.4 0.65 68.3 70.8 0.009 - - - - -E5-2630 118 32 30.9 221.9 0.27 - - - - -X5560 524.5 1.05 521.2 525.4 0.002 - - - - -

Table 2: Different workloads‘ runtimes over various CPU architectures and containers [13], [32], [38]

Containerisation over Virtualisation: The lack of isolation and efficient resource sharing withresource over-subscription (soft limits) makes running containers inside VMs a more feasible ar-chitecture [as happens in AWS EC2 container service, Lambda that uses Dockers and Google


container engine]. Soft limits enable applications to use resources beyond their allocated limits ifthose resources are not in use (over-subscription). For VMs, resource limits are usually hard whichmeans that VMs are not allowed to utilize more resources than their provisioned resources even ifthe resources are idle (no over-subscription). Soft limits and over-subscribing resources (idle) mayprovide more efficient resource utilization and management [39]. Merging containers with VMscould provide the benefits of containers along with benefits of VMs. For example, VMs are securethan containers and containers could provide better resource utilisation levels than VMs. Fur-thermore, Sharma et al. [36] stated that containers in VMs provide performance benefits as well.The authors suggest that neighbouring containers within a VM can be trusted because containersfrom a single tenant may be allowed to run in a particular VM. Mondesire et al. [40] suggest thatrunning containers inside VMs instead on bare-metal hardware have at least two advantages: (i)if a particular container needs restarting, only the VM, which accommodate this container, willbe restarted without affecting other VMs on the same host; and (ii) snapshots of VMs could bemigrated to other hosts, if needed.Table 3 show variations in runtimes when various workloads or functions are being executed invirtualised containers (containers run inside VMs) over different CPU architectures. These statis-tics were collected from various published works [13], [32], [38]. Using monte-carlo simulationsfor these statistics (with respect to mean and standard deviation), we observed that containersrunning inside VMs could increase resource utilisation; however, the workload performance is neg-atively affected, probably, due to large number of co-located containers [41]. Moreover, for certainworkloads, virtualised containers may provide for comparable performance to bare-metal; however,for other workloads, the reverse might be true as their performance is lower than VMs. This isillustrated visually in Fig. 8.

Workload CPU model single vCPU two vCPUs(µ) (σ) Min Max CoV (µ) (σ) Min Max CoV

E5-2665 284.2 1.45 279.8 288.7 0.005 - - - - -Pxz E5540 683.8 2.8 674.7 695.4 0.004 388.9 3.8 375.8 402.1 0.01

X5650 900 3.2 889.7 909.2 0.004 - - - - -Bzip2 E5-4650 - - - - - - - - - -

i7-4600U 681.3 68.1 455.2 899.9 0.095 - - - - -E5-2630 621 23 535.1 687.9 0.037 - - - - -X5560 1,634 26 1,584 1,688 0.016 - - - - -

———————————————————————————————————Stream E5-2665 62.2 1.33 58.3 65.9 0.022 - - - - -

E5540 211.3 2.1 205.3 219.9 0.01 131.4 2.1 124.7 138 0.016Povray X5650 1,491 6.9 1,468.0 1,511.3 0.000 - - - - -

E5-4650 - - - - - - - - - -E5-2630 149 2 142.6 155.8 0.013 - - - - -X5560 527 0.5 526 528 0.000 - - - - -

Table 3: Different workloads‘ runtimes over various CPU architectures and virtualized containers[13], [32], [38]

Bare-metal: The business requirements of organisation may include full and privileged access toraw hardware which they provision for their services. This may also happen for certain workloadstypes, such as HPC and real-time applications, that need the best performance. Therefore, it wouldbe essential to run user’s application on bare-metal hardware instead of packing it in a containeror a VM. Amaon AWS has recently launched bare-metal instance class which offers complete ac-cess to the provisioned resources (the recent M5 instances). Similarly, the Rackspace cloud also


offers bare-metal hardware known as “OnMetal”. Moreover, bare-metal offers several advantagesover containers and VMs, for example: (i) security - multiple containers|VMs on a single VM|hostcreate chances for network attacks such as denial of service; and (ii) not just security, but evenperformance is affected - the “noisy neighbour” or co-location problem. In the literature, vari-ous researchers have characterized the performance of containers, VMs, virtualised containers andbare-metal; and in all cases, the later one performs better than the former ones [31], [32], [36]. Asshown in Table 4, variations in runtimes can be seen across various CPU platforms (architectures)for various kinds of applications and hardware platform (VMs, containers, container over VMs,bare-metal). Several reasons, such as CPU contention, cache design, for these kinds of variationsover bare-metal hardware are described in [41].

Workload CPU model single core two cores(µ) (σ) Min Max CoV (µ) (σ) Min Max CoV

E5-2665 290.8 1.13 287.6 294.4 0.004 - - - - -Pxz E5540 670.8 6.9 647.7 694.4 0.010 360.4 4.3 343.4 376.4 0.013

E5-2620 16.7 3.83 6.4 28.6 0.226 - - - - -Bzip2 X5650 811 4.1 799 824.3 0.005 - - - - -

E5-4650 - - - - - - - - - -AMD 6230 13.7 0.03 13.6 13.8 0.002 - - - - -E5-2630 418 37 307.8 518.2 0.088 - - - - -X5560 1,600 23.75 1,575 1,670 0.015 - - - - -

———————————————————————————————————Stream E5-2665 76.2 0.93 73.2 79.1 0.012 - - - - -

E5540 192.0 0.8 189.6 194.3 0.004 109.5 1.0 106.4 112.8 0.009E5-2620 87.24 1.4 82.7 91.8 0.016 - - - - -

Povray X5650 1,405 4.8 1,390.4 1,418.4 0.000 - - - - -E5-4650 - - - - - - - - - -

AMD 6230 68.5 0.02 68.4 70.8 0.000 - - - - -E5-2630 100 10 61.9 134.8 0.101 - - - - -X5560 521.6 0.625 520.4 522.9 0.001 - - - - -

Table 4: Various kinds of workloads and their runtimes (in seconds) when run over various bare-metal hardware with different CPU architectures [13], [32]

In Tables 1, 2, 3, 4, some statistics such as mean, minimum (min) and maximum (max ) were beingcalculated manually; as they were not available in the literature. For example, where min and maxvalues were not found but mean (µ) and standard deviation (σ) were present, we used monte-carlosimulations and statistical distributions (log-normal for VMs and normal for others) to calculatethem from the µ and σ. Moreover, where µ and σ were not found, we used range equation tocalculate them from the min and max. Fig. 6, 7 and 8 show the performance (runtimes) of vari-ous applications, when run over various sand-boxing technologies, on various CPU architectures.Large variations can be seen for Povray benchmark on X5560 as compared to Bzip2. Moreover,containers could be of comparable performance to bare-metal; however, in certain scenarios theirperformance may be even worse than VMs and containers|VMs both. We speculate this situationmight happen if there is a large number of containers co-located on a single hosts, with similarworkloads competing for same resources. Fig. 8 illustrates that containers could offer comparableperformance to bare-metal; however, bare-metal consume more energy due to lower resource util-isation levels. Moreover, for certain workloads, the performance of virtualised containers may beworse than VMs.


Fig. 6: Variation in runtimes of two different applications (left: Bzip2 – right: Povray), when runover different sand-boxing technologies, on X5560 CPU platform [Bzip2 in containers and bare-metal has comparable runtimes; Povray in containers performs better than VMs and virtualisedcontainers]

Fig. 7: Variation in runtimes of two different applications (left: Bzip2 – right: Povray) when runover different sand-boxing technologies, on E5-2630 CPU platform [both applications, when run invirtualised containers, may perform better than VMs and containers; containers could be as badas best in certain scenarios]

Fig. 8: Variation in runtimes of two different applications (left: Bzip2 – right: Povray) when runover different sand-boxing technologies, on E5-2665 CPU platform [both applications, when run invirtualised containers, may perform better than VMs and containers; containers could be as badas best in certain scenarios]


6 Performance Evaluation

Bin-packing problems are usually solved using various heuristics that might not guarantee opti-mal results, however, they are enough fast to address large-scale problems [15]. We assume thescheduling problem as moving from a given datacenter state to an ideal state, which should beone using the fewest and more energy, performance efficient hosts. We achieve a datacenter stateby implementing the HeporCloud approach, with consolidation approach that guarantees energyand performance efficiencies. To evaluate the effect of this, we consider: (i) no migration; (ii) mi-grate all; (iii) migrate individual workloads either from bare-metal, VMs, containers or containersover VMs; and (iv) HeporCloud which predicts an effective migration among various migrationspossibilities. Moreover, we consider various scheduling policies. Furthermore, we also investigatemigrations performed within a particular platform (inter-platform) and those triggered across var-ious platforms (intra-platforms).We assume the consolidation (migration) process as an optimization problem with the objectiveto decrease the number of hosts in use. Every 5 minutes’ interval, the optimization module isrun grounded on the present utilization levels of all hosts, in three steps; (1) containers selection:Every host is monitored and if its present utilization level is less than a pre-defined Thresholdlow(lower threshold value e.g. 20%), all accommodated VMs and containers on this particular hostare chosen for migration. If there are several VMs and containers suitable for migration, then thesuggested container selection policy [Alg. 2] gives priority to the VM or container that have moremargins for cost savings (Psavings) – (we prefer to migrate one VM or container at a time froma particular host to minimize performance loss); (2) hosts selection: The migration policy choosesthe most suitable host from all available hosts that could run these VMs and containers energy andperformance efficiently. Nevertheless, to decrease the number of active hosts, it avoids allocationto: (i) switched off hosts (if it is possible); and (ii) hosts that intend to go into idle|switched offstate (switched on but with no work on them|idle); and (3) placement: The list of selected VMs andcontainers is sorted in decreasing order of their Rpredicted (future runtimes) that favours to migratelong-running VMs|containers first. Finally, a particular VM or a container allocation algorithm isused to reallocate all VMs and containers, as placement of migratable entities is a sub problem ofthe consolidation with migration process [8].

Evaluation Metrics: We consider total number of migrations (inter-platform and intra-platforms),energy consumption (KWh) and workload performance (execution time measured in minutes) asthe performance evaluation metrics.

6.1 Experimental Setup

The experiments were performed in an event driven simulation platform through merging bothCloudSim [18] and ContainerCloudSim [9]. CloudSim provides support for virtualisation (VMs)and ContainerCloudSim is known for containers. However, ContainerCloudSim does not have thecapability to run containers over VMs. Moreover, its currently available version does not supportVMs migration, however, a bit or partial implementation of the optimisation module can be seenin the existing code. We have extended its various classes, such as the broker and the optimisationmodule, in order to perform our experiments and evaluation. The extended broker has the capabil-ity to start a VM/container, at runtime, when a request is received that matches task arrival timesin the Google cluster dataset. A cluster (simulated using CloudSim) of 12,583 heterogeneous hosts,which consists of various architecture types (with respect to varying performance) and hardwarespecifications – as given in Table 5 - is available to run various kinds of benchmark workload. Thesimulated hosts are configured based on several reasonable assumptions as described in Sec. 5.The specification (of hardware) and energy consumption values for the hosts were collected fromthe well-known SPECpower5 benchmarks. The hosts’ energy consumption, migration costs andperformance of VMs and containers are assumed as described in [8]. Moreover, we used four kinds

5 https://www.spec.org/power ssj2008/


of workloads, i.e. Hpc (bare-metal), W1 (VMs), W2 (containers) and W3 (virtualised containers),that belong to real data from major cloud providers i.e. Intel’s compute cloud [5], Google [7] andMicrosoft Azure [6] clusters, respectively.

CPU SPEED NO OF NO OF MEMORY PIDLE PMAX AMOUNTMODEL (MHz) CORES ECUs (GB) (Wh) (Wh)

E5-2630 2,300 12 27.6 128 99.6 325E5430 2,830 8 22.4 16 166 265E5507 2,533 8 20 8 67 218

E5-2620 2,000 12 24 32 70 300E5645 2,400 12 28.8 16 63.1 200

E5-2650 2,000 16 32 24 52.9 215 12,583E5-2651 1,800 12 21.6 32 57.5 178E5-2670 2,600 16 41.6 24 54.1 243E5540 2,500 4 10 72 151 312X5560 2,800 8 22.4 128 133 288

E5-2665 3,000 8 24 256 117 314X5650 2,666 12 31.2 64 80.1 258

Table 5: Host characteristics for Amazon’s cloud

Our simulation environment consists of six types of VMs which resemble to Amazon’s instanceclasses as given in Table 6. The VM types are further ranked (in terms of resource capacities andperformance) according to Amazon’s description of their VM performance rating – ECU (the EC2Compute Unit), which is described as: “equivalent CPU capacity of a 1.0 – 1.2 GHz 2007 Opteronor 2007 Xeon processor”; and its variations in performance is suggested approximately 20% (1.0 –1.2 GHz) in [33]. Since, the above ECU rating is per core, therefore, the total rating of a particularhost (with multiple cores) can be calculated by multiplying its total number of cores and the ECUrating [33].

Instance No of No of Speed Mem Storagetype vCPUs ECUs (GHz) (GB) (GB)

MIPS

t2.nano 1 1 1.0 0.5 1t1.micro 1 1 1.0 0.613 1t2.micro 1 1 1.0 1 1m1.small 1 1 1.0 1.7 160

m1.medium 1 2 2.0 3.75 410m3.medium 1 3 3.0 3.75 4

Table 6: Amazon different instance types and their characteristics – Mem means memory (RAM)

We assume that these hosts and VMs are comparable by a single measure that permits for perfor-mance ranking, for which we assume the CloudSim’s notion of “Million of Instructions Per Second(MIPS)” as a proxy. One technique to VM sizing is to assign a VM as a single core for the max-imum value 1, half a core (hyperthread) for 0.5, and consider that higher VM gearing leads to aquarter of a core for 0.25. But to address allocation more flexibly, along lines with particular IaaSproviders, we map CPU frequency for the hosts given to Amazon’s ECUs as: 1 GHz CPU, 1.7GBRAM, giving numerous instance types6 [8]. Further, the ECU maps MIPS for consistency with the

6 http://www.ec2instances.info


CloudSim simulator (Table 6), and we consider that each instance requires, at least, 1 ECU and 1vCPU (core) or more, as given in Table 6. The speed of every instance type (MIPS rating) is themultiple of number of ECUs (1 ECU = 1GHz) and vCPUs (cores). For instance, the speed of them1.medium instance, as shown in Table 6, is 2 (ECUs) × 1 (vCPU) = 2 (GHz).To address a cloud context such as Amazon Lambda, each Google’s task is assigned a single, no-tional, container, as shown in Table 7, that corresponds to Google machine types with the onlyexception that all the containers are single core. Similar to VM sizing, containers are also sized(MIPS) and each VM can accommodate more than one containers. For example, a m3.mediuminstance can accommodate three containers of type A, while two containers of type C and so on.Each task utilizes its allocated resources according to task statistics and usage in Intel’s cloud [5],Google [7] and Microsoft Azure datasets [6].

Container Speed Cores ECU’s Memorytype (MHz) (MB)

A 1,000 1 1 128B 1,225 1 1.23 256C 1,500 1 1.5 512

Table 7: Container types and their characteristics

Each container is assumed to run four workload types that belong to either Intel’s cloud [5],Google’s cluster [7] or Microsoft Azure [6] datasets. These datasets denote four various kinds ofworkloads i.e. HPC (bare-metal), containers, VMs, containers|VMs, respectively. Due to the un-availability of the fourth dataset i.e. virtualised containers (when containers run inside VMs), weused a synthesized workload which was derived from various statistics such as mean (µ) and stan-dard deviation (σ), as shown in Table 3. Further, the utilization of each workload type is modelledas a normal distribution function over the mean CPU usage, as calculated from the original traces,at 5 minute intervals. Moreover, the total execution time of each application is the sum of its everytasks‘ execution times. Each dataset consists of 6,000 tasks whose sum of execution times are 43.22,68.67, 82.41 and 179.8, in hours.Initially, all containers and VMs are allocated according to: (i) the resource requirements definedby the container|VM types; (ii) container allocation policy; and (iii) VM allocation policy. Notethat all containers|VMs are allocated to VMs|hosts using the workload-aware First Fit (FF) al-location policy and their arrival correspond to task arrival times and rate in Google and Azureclusters datasets [8]. However, using the workload data, if containers|VMs utilize their provisionedresources less, this create opportunities for consolidation. Note that each container|VM is randomlyassigned a workload trace from one of the instances (Intel+Azure+Google+Synthesized datasets)that runs until its execution is completed. During consolidation, the optimization module deter-mines the under-loaded and overloaded hosts using two techniques: (i) the single threshold policy– which uses a static upper utilization threshold (80%) for each host; and (ii) double threshold’spolicy – which uses a lower (20%) and an upper (80%) threshold values. In respect of the resultspresented in this paper, we use the later approach i.e. two threshold values. If there are severalmigratable containers|VMs, then the migration policy uses their rating [the product of energy andruntime is described in Sec. 4] to prioritize those containers|VMs that will save more energy whilemaximum performance is maintained (Psavings). Moreover, target hosts|VMs for the migratableVMs|containers are identified using a modified version of the default host selection policy in Con-tainerCloudSim [9]. The modified host selection policy accounts for host’s ERP in order to pickan energy and performance efficient one for the migratable entity.We made a slight modification to the above experiments in order to account for the migration cost(in terms of energy consumption and performance loss). The VM migration time can be calculatedfrom the size of the VM and the available bandwidth, plus the time needed to start the new VM.Previous works [9], [18] have assumed that half of the total bandwidth is available for VM migra-


tion, while the other half is reserved for VMs communication. Other works [32], [42] assume thatthe container migration time is the amount of time needed to start a new container at the targethost. If an already available VM cannot accommodate the container, then a new VM is createdfirst – in that case, the container migration time is the sum of starting a container and a VM plusthe memory pages dirtied by the container. For simplicity reasons, they further assume that thecontainer is stateless, and there are no dirtied pages which needs to be copied. In our experiments,the migration total time and down time are taken from [43], which are experimentally evaluated,as shown in Table 8. Moreover, consolidation can bring several hosts to idle states, thus makingit possible to switch off (bring to sleep states) them in order to save energy. Switching on/off andstates transitions of hosts can bring scheduling delays. We assume container|VM start, reboot, offtimes as shown in Table 9. These costs (delays) are very important, in the context of cloud com-puting, as users are charged for the total duration of their service [workloads runtimes + instanceslaunch times]. These values are also used inside ContainerCloudSim [9], by default. Furthermore,containers|VMs performance is affected as described in Sec. 5. Transitions among various states ofa host also incur non-negligible energy overheads [8].

Workload 66% utilisation 100% utilisationmean std. dev. min max mean std. dev. min max

KVMDowntime 14 1.886 12 17 16 2.404 13 21.5

Migration time 119.1 3.281 115 125 129.8 3.553 125 135

LXCDowntime 8 1.155 6 10 10 1.886 7 12.5

Migration time 82 3.266 77 90 95.7 1.703 92 98

Table 8: Virtualisation and containerization total migration-time and down-time (in seconds) atvarious workloads with different utilisation levels [43]

Container VM Container|VM Host

Start 1.623 3.005 4.628Off 2.493 64.422 66.915

Restart (soft) 4.209 125.463 129.672 600 [13]Restart (hard) 4.338 6.043 10.381

Delete 2.473 3.767 6.24

Table 9: Configuration times in seconds [ProLiant DL580 Gen8 with Ubuntu Server 14.04] – thehost configuration energy consumption is computed based on its maximum power consumptionand the transition times from one state to another [8], [39]

6.2 Experimental Results

The simulated cloud environment is composed of 12,583 hosts, 3,800 containers|VMs with configu-ration shown in Table 5, Table 6 and four kinds of workloads that belong to bare-metal, virtualised,containerised and virualised containers, respectively. When a container|VM request is received, acontainer|VM is created from a list of available flavours as shown in Table 7, Table 6 and is placedon a suitable host or a VM which is already running on a particular host. The scheduler is work-load aware which predicts the type of workload; then the platform which might offer performancebenefits; and then a most energy and performance efficient host to run the workload. Moreover,


the scheduler uses a First Fit (FF) heuristic to place similar workloads in a particular platform.If there is no suitable VM to accommodate the container, then a new VM is created from a listof available instance types as shown in Table 6. Various container types are described in Table7. We assume that the container workload is heterogeneous and, therefore, changes when a con-tainer is being migrated from one host to another. For bare-metal, the container is assigned solelyto a particular host; and no other allocation is possible to this host until the container is beingterminated and the host resources are free. The allocation policy is workload-aware that allocatecontainers to appropriate platform. After each 5 minute interval, the HeporCloud technique checksfor consolidation opportunities, and selects effective migrations from a list of possible migratableentities.

6.3 Results Discussion

The results of various migration policies are shown in Table 10 (total energy consumption) andTable 11 (execution time). With slight decrease in performance from 2.14% to 3.02%, the proposedHeporCloud framework could significantly decrease the total energy consumption, that could beup to 30.47%, and the total number of migrations. Large variations in energy consumption forthe migrate all approach demonstrate that uncontrolled migrations could lead to datacenter ineffi-ciency. In our experiments, we found that efficient allocation could be approximately 10.18% moreenergy efficient than the no migration technique. Similarly, an overlap between inter-platform andintra-platforms migrations shows an existing trade-off that is, largely, dependent on the workloadtype and datacenter configuration. From performance perspective, intra-platforms migrations aremore effective than the inter-platform migrations; if and only if migrations are triggered to moreperformance efficient hosts. Otherwise, if migrations are uncontrolled then intra-platforms migra-tion are worse than inter-platform migration.

Policy Bare-metal VMs Containers Containers|VMs HeporCloudNo migrations 2,563 2,678 2,792 2,934 2,412 [±121]Migrations ——————————————————————————Migrate all 2,982 2,856 2,921 2,785 2,353 [±437]

HeporCloud ——————————————————————————inter-platform 2,456 2,087 2,873 3,173 1,782 [±57]intra-platforms 2,389 2,129 2,666 3,198 1,759 [±98]

Table 10: Energy consumption in kWh [the values followed by± symbol denote standard deviations]- the overlap for no migration and migrate all approaches under HeporCloud field represents thetrade-off between efficient allocation and migration

An interesting behaviour can be observed when using a single (centralised) scheduler and dis-tributed (individual) schedulers, as shown in Table 12. For example, when migrations are nottaken into account, single scheduler is approximately 7.62% – 16.86% more energy efficient thanthe distributed schedulers. However, when migrations are considered, then: (i) for uncontrolledmigrations – the single scheduler produces large variations in energy consumption; and (ii) forcontrolled migrations – the distributed schedulers produces large variations in energy consump-tion. In both cases, using a single (centralised) scheduler is more energy efficient than using thedistributed, but, individual schedulers for each sand-boxing platform. Moreover, the total numberof triggered migrations and, therefore, performance of various applications is affected when a singlescheduler or several distributed schedulers are used. Sine, the single scheduler has knowledge of allplatforms and resources; therefore, more performance aware migration decisions can be triggeredas compared to distributed schedulers.As shown in Fig. 9, the HeporCloud technique is approximately 14.61% – 37.97% more energy


Migration Workload Sand-boxing technology Prediction accuracypolicy type Bare-metal VMs Containers Containers|VMs HeporCloud Workload Resource

Hpc 43.22 67.32 51.52 65.83 43.22 - -W1 66.89 72.77 69.8 73.1 68.67 - -

No W2 71.43 98.19 87.71 91.88 82.41 - -W3 173.2 211.8 198.2 191.9 179.8 - -

HeporCloud 170.91 207.29 199.11 190.87 172.77 59.8% 78.8%inter-platform migrations

Hpc 44.98 67.32 51.52 66.32 43.22 - -Hpc W1 69.12 72.77 69.8 74.9 68.67 - -

W2 76.89 98.19 87.71 94.76 82.41 - -W3 147.1 172.3 168.9 167.9 155.6 - -Hpc 43.22 78.86 51.52 55.56 43.22 - -

W1 W1 66.89 79.74 69.8 80.8 68.67 - -W2 71.43 99.14 87.71 99.12 82.41 - -W3 147.1 172.3 168.9 167.9 155.6 - -Hpc 43.22 67.32 52.67 69.54 43.22 - -

W2 W1 66.89 72.77 70.1 69.98 68.67 - -W2 71.43 98.19 87.93 88.34 82.41 - -W3 147.1 172.3 168.9 167.9 155.6 - -Hpc 43.22 67.32 52.67 69.54 43.22 - -

W3 W1 66.89 72.77 70.1 69.98 68.67 - -W2 71.43 98.19 87.93 88.34 82.41 - -W3 141.9 170.1 166.9 167.0 151.4 - -Hpc 44.99 68.89 52.56 65.99 44.67 56.1% 43.23%

Migrate all W1 69.78 72.01 68.88 75.31 68.12 60.4% 44.8%W2 77.17 98.79 88.34 93.97 84.09 58.9% 46.2%W3 149.87 177.11 169.88 177.55 159.11 59.7% 47.8%Hpc 43.29 68.21 53.78 56.67 45.22 60.1% 79.9%

HeporCloud W1 67.34 72.89 70.3 68.12 68.97 62.2% 80.2%W2 71.67 99.7 88.01 81.07 72.99 58.9% 80.3%W3 141.9 170.1 166.9 167.0 151.4 59.5% 79.5%

intra-platforms migrationsHpc 51.91 69.23 55.89 60.78 49.32 - -W1 66.99 71.56 73.67 69.32 65.21 - -W2 74.32 94.12 89.45 82.78 74.1 - -W3 139.23 169.89 165.54 164.87 153.23 - -

Migrate all 141.76 168.12 164.32 165.89 155.99 61.45% 48.8%HeporCloud 138.89 166.23 165.23 165.98 148.23 62.98% 84.8%

Table 11: Performance of various workloads using different techniques to migrations and virtualisa-tion (in minutes) [the workload prediction accuracy represents scheduling and resource predictionaccuracy corresponds to the efficiency of hosts]

efficient and 7.23% – 20.0% more performance efficient than only virtualisation and containeriza-tion technologies, respectively. When migrations are disabled, our approach is ∼6.3% more energyefficient than the bare-metal. However, the savings can be up to 37.8%, if migrations are takeninto account. These savings for virtualised containers were observed as 17.8% and 43.8%, respec-tively. Moreover, our evaluation suggests that bare-metal provides optimal performance; however,the energy consumption is strongly dependent on the type of workload and job arrival patterns.Although, a single job can be executed much quickly that could decrease the energy consumption


Policy Distributed scheduler Single scheduler Improvement (%)No migrations 2,742 [±159] 2,412 [±121] 7.62 – 16.86

MigrationsMigrate all 2,886 [±85] 2,353 [±437] 0.39 – 20.8

inter-platform 2,647 [±475] 1,782 [±57] 15.33 – 44.75intra-platforms 2,596 [±458] 1,759 [±98] 13.14 – 45.61

Table 12: Energy consumption (in kWh) using single and distributed schedulers [Improvementmeans using single (centralised) scheduler instead of distributed (individual) for all platforms]

of that particular job; however, the provisioned resources are not available to other jobs waitingin the queue. Furthermore, bare-metal resources are largely under-utilised. This increases job waittimes and, therefore, negatively affects the workload performance, user cost and datacenter energyefficiency.

Bare-metal VMs Containers Containers|VMs HeporCloud0

500

1000

1500

2000

2500

3000

3500

En

erg

y C

on

sum

pti

on

(K

Wh

)

No migrationConsolidation

Bare-metal VMs Containers Containers|VMs HeporCloud0

10

20

30

40

50

60

70

80

90R

un

tim

es (

ho

urs

)

No migrationConsolidation

Fig. 9: Energy consumption (KWh) [left] and overall performance (execution times in hours) [right]

Similarly, VMs provide for high energy consumption due to poor performance, particularly, whichmay happen due to co-location. When containers run inside VMs, their performance is limited tothe VMs capacities and resources – consequently increasing energy consumption. The approach“bare-metal i.e. containers run directly on bare-metal” provides for optimal energy and perfor-mance efficiencies. When the allocation is aware of the workload type, then optimal energy andperformance efficiency could be achieved.When migrations are considered, the proposed technique provides for greater energy efficiency atcomparable cost of performance to the “bare-metal” approach. Moreover, as the orchestrator hasknowledge of all the four platforms, it can trigger energy and performance efficient migrations [4].Unlike other published results [38], our evaluation suggests that bare-metal could be expensive interms of energy consumption, probably due to the lowest resource utilisation levels [13]. Containers,when run inside VMs, provide the highest level of resource utilisation; however, their performanceis limited to the performance of VMs – thus consuming more energy than VMs and containers,as shown in Fig. 9. Another reason for this low performance of virtualised containers is, probably,the large ration of co-location and neighbouring containers that might compete for same resources.This suggests a trade-off between VMs and containers (when run inside VMs) with respect toenergy consumption and workload performance.Table 13 shows the migration statistics such as the percentage of migratable entities, those whichwere able to recover their migration costs, and the accuracy of the prediction technique. The datashow that HeporCloud has significantly reduced the total number of migrations and, largely, migra-tions have recovered their costs. We observed more number of triggered migrations across platforms


Migration Migratable Cost recovered Runtimepolicy % % accuracy

NO - - -Migrate all inter-platform 4.01 (±2.1) 24.87 (±2.8) 78.45% (±1.01)HeporCloud 3.17 (±1.4) 79.02 (±4.2) 79.33% (±1.11)————————————————————————————————

NO - - -Migrate all intra-platforms 6.87 (±2.3) 41.76 (±3.8) 69.41% (±1.88)HeporCloud 5.01 (±1.2) 75.89 (±2.6) 70.34% (±2.26)

Table 13: Percentages of migratable entities, cost recovery and prediction accuracy [the sign ±denotes standard deviation over various runs]

(intra-platforms) as compared to with a particular platform (inter-platform). The prediction ac-curacy is computed through analysing the data we gathered during numerical simulations. Forexample, in case of inter-platform migrations, approximately 79.33% of the migrated entities werecontinued to run until they recovered their migration costs toff . In other words, the predictionaccuracy does not reveal that how accurately runtimes were estimated; instead, it reveals that howmany runtimes were lengthier than the toff ; and the migratable entities were able to recover theirmigration costs.

6.4 Impact of Datacenter Configuration on Energy and Performance

How hosts are addressed in a datacenter (configuration) has an impact on the resource allocationapproach. For example, if hosts are in increasing order of their energy efficiencies (efficiency factor– Ef ) and a particular algorithm is used to allocate these resources; then, the total energy con-sumption would be different if hosts are arranged in decreasing order of their energy efficiencies.Similarly, if hosts are ordered based on their performance (CPU models), or both energy consump-tion and performance (ERP ), then the existing trade-off between energy and performance wouldalso differ for various resource allocation, consolidation (migration) policies and workloads. Wediscuss the impact of host ordering and allocation approaches on infrastructure energy efficiencyand workload performance, therefore, costs. Each resource allocation and migration policy selectsa particular host to execute the given workload; where the starting point for such decisions couldproduce variations in energy consumption, performance and, therefore, users monetary costs. Forexample, if the initial ordering were reversed, this may change the experimental outcomes in termsof energy consumption and workload performance [8].

NO Migrate all HeporCloud

1800

2000

2200

2400

2600

2800

En

erg

y co

nsu

mp

tio

n (

KW

h)

NO Migrate all HeporCloud1600

1800

2000

2200

2400

2600

2800

En

erg

y co

nsu

mp

tio

n (

KW

h)

NO Migrate all HeporCloud1000

1500

2000

2500

3000

3500

4000

En

erg

y co

nsu

mp

tio

n (

KW

h)

Fig. 10: Variations in energy consumption (KWh) with respect to various migration policies anddatacenter configurations [left: NO – middle: DEC – right: INC]

To determine the impact of datacenter configuration on energy consumption and workload per-formance, we run the experiments from Sec. 6.1 ten times with three different initial orders for


hosts: (a) no order (NR) – random; (b) increasing order based on Ef (INC); and (c) decreasingorder based of Ef (DEC). The Ef for each host is computed through dividing its maximum powerconsumption (Pmax – energy consumed at 100% utilisation level) by the number of slots (coresor GCEUs) it has. For example, if we have four hosts (H1, H2, H3 and H4) having EH1

f = 4,

EH2

f = 1, EH3

f = 2 and EH4

f = 3; where smaller Ef represents lower energy efficiency. Then thecorresponding orders would be: NR – [H1, H2, H3, H4], DEC – [H1, H4, H3, H2] and INC – [H2,H3, H4, H1]. We can also use other metrics such as ERP to compute order for hosts. Various ordersof hosts create various levels of energy and performance efficiencies. Fig. 10 and Fig. 11 demon-strate variations in energy consumption and workload performance (runtime) for various kinds ofworkloads and hosts orders. Here, ordering is discussed in terms of allocation policies that meanslogical addressing and is not a physical shift. However, this might be extended to: (i) putting hostsin different racks; and/or (ii) physically shifting hosts inside a rack. This demonstrates that thephysical order of hosts does matter and could affect energy efficiency and workload performance(hence cost) in large datacenters.


140

150

160

170

180

Exe

cuti

on

tim

e (m

inu

tes)


120

130

140

150

160

170

Exe

cuti

on

tim

e (m

inu

tes)


130

140

150

160

170

180

Exe

cuti

on

tim

e (m

inu

tes)

Fig. 11: Variations in workload performance (minutes) with respect to various migration policiesand datacenter configurations [left: NO – middle: DEC – right: INC]

6.5 Costs Savings

The total electricity bill, user monetary costs and costs savings (in US dollars - $) are described inTable 14. For these analysis, we assume a PUE7 of 1.10 and energy price of 0.88$ per KWh8 thatmimic a Google datacenter located in the Oklahoma state, USA. Moreover, we assume that theusers bills are computed at 0.0017$ per second9. The providers could save up to 35.9% energy costsusing the HeporCloud technique. Moreover, the users costs could be reduced up to approximately7.91 to 14.2%.

Policy Energy Users monetary Total costscosts ($) costs ($) savings (%)

No migrations 2654.26 1057.35 -Migrate all 2793.65 973.75 7.91HeporCloud 1702.7 907.17 14.2

Table 14: Costs savings [Energy and users monetary costs are described in US dollars]

7 https://www.google.co.uk/about/datacenters/efficiency/

8 https://www.eia.gov/electricity/monthly/

9 https://aws.amazon.com/ec2/pricing/


6.6 CIAO vs. HeporCloud Framework

The results in terms of energy consumption and workload performance are shown in Table 15 andTable 16, respectively. The HeporCloud orchestrator is approximately 9.4% – 14.2% more energyand 1.7% – 17.9% more performance efficient than the Intel’s CIAO framework when several com-binations of workloads and migration policies are taken into account. We also observed a similarbehaviour of the CIAO framework; due to the existing trade-off between energy consumption andworkload performance [14]. Using several reasonable assumptions, our evaluation suggests thatapproximately 13.57% energy could be saved at cost of 3.88% loss in performance. These savingsseem to be in line with possible savings using the HeporCloud framework i.e. ∼30.47% energysavings at cost of 2.14% loss in performance.

Policy CIAO HeporCloud ImprovementNo migrations 2,662 2,412 9.4%Migrations ———————————————

inter-platform 2,076 1,782 14.20%intra-platforms 2,076 1,759 15.27%

Table 15: Energy consumption of the CIAO and HeporCloud architectures in KWh [minimumvalues are “best”]; these improvements can be translated to providers energy bills

Workload Performance Improvementtype CIAO HeporCloudHpc 55.09 45.22 17.9%W1 70.18 68.97 1.7%W2 86.46 72.99 15.6%W3 170.91 151.4 11.42%

Table 16: Performance comparison of the CIAO and HeporCloud architectures – performancemeans workload execution time in minutes [the “best” results are shown in boldface]; these im-provements can be translated to users monetary costs

These results demonstrate the efficiency of our proposed HeporCloud framework. Due to the exist-ing trade-off between energy consumption and workload performance, it is impossible to improveboth; however, the best approach would improve one factor with a slight decrease in another fac-tor. The trade-off can be observed accurately using a single metric i.e. Energy Runtime Product(ERP )10 – the product of energy consumption (P) and performance (R); where runtime is theinverse of performance (as lower workload run-times mean good performance and vice versa). Adetail discussion of the ERP metric is given in Sec. 3. Table 17 describes ERP values for severalmigration policies along with the CIAO and HeporCloud frameworks. In case of “no migration”technique, albeit performance is optimal (minimal); however, energy consumption is maximum.For uncontrolled migrations (Hpc, W1, W2, W3), variations in energy consumption and workloadperformance can be seen; along with significant overlaps, which represent that migrations couldbe expensive than the “no migration” approach. In such scenarios, workload-specific resource al-location (for example HeporCloud) could offer energy savings and performance guarantees. Theproposed HeporCloud approach has an ERP of 80.6 which is far better than the CIAO’s ERP of114.4.

10 http://epubs.surrey.ac.uk/841959/


Policy Energy (E) Performance (P) ERPE × P

No migrations 2,662 44.89 119.5Hpc 2,221 55.09 122.4W1 2,189 70.18 153.6W2 2,299 86.46 198.8W3 2,311 151.4 349.9

CIAO 2,076 55.09 114.4HeporCloud 1,782 45.22 80.6

Table 17: Energy Runtime Product (ERP )10 to demonstrate the trade-off between energy con-sumption and performance using CIAO and HeporCloud frameworks [minimum values are “best”]

7 Related Work

It is notable to mention that Intel has proposed a system i.e. CIAO, which operates VMs and con-tainers through the same system. However, Intel do not offer too many details nor publishing anyofficial description. Albeit, the CIAO’s project code is available for free on-line; however, certaindetails are still unknown. For example: (a) how the scheduler makes workload specific decisions; (b)how container and VMs are being migrated; and (c) there is no published manuscript that demon-strates its evaluation in terms of workload performance and energy efficiency. Kominos et al. [13]have discussed bare-metal, VMs and containers (individually) in terms of performance, network,memory, disk I/O (input output) and boot time (seconds). However, resource management with re-spect to consolidation, energy efficiency and workloads (applications) or resources heterogeneitiesare not explored. Lubomski et al. [44] have compared various virtualisation techniques such asVMs, containers; and suggested non-significant performance loss when containers run within VMs.However, Mavridis et al. [45] suggest that the performance loss could be significant.The Intel’s CIAO architecture, as shown in Fig. 12, consists of: (i) a controller which is responsibleto communicate with the users; (ii) a scheduler that allocate resources to user’s workload; and (iii)a launcher which gathers processing statistics at each compute host. The controller and schedulerrun on one system, while the launcher runs on every compute server in the cluster. Furthermore,the scheduler uses the launcher statistics for workload specific placement decisions. The scheduleruses a first fit (FF) heuristic technique to allocate resources rather than the best fit (BF) approach.The Intel’s researchers bias for dispatching speed and implementation simplicity over an absoluteoptimality. However, a BF, or other similar heuristics, might be essential for better chance of suc-ceeding the placement of future, unknown, workloads. Moreover, the scheduling choice focuses,primarily, on memory, disk and CPU availability of various compute nodes relative to the start ofrequested workload11.The CIAO architecture does not support live migrations; however, through check-pointing i.e.stopping and restating, a particular instance could be migrated from one host to another host,seamlessly. Moreover, several instances running on a particular host can also be migrated, concur-rently, through a single command. This might be useful if a host needs to be temporarily removedfrom the cluster for maintenance and up-gradation purposes. The CIAO project documentationalso suggests that the scheduler currently implements a trivial approach which prefers not usingthe most-recently-used (MRU) compute host for workload placement. Although, this could be in-expensive and may lead to enough spread of new workloads across the cluster; however, this maynot be essentially energy efficient. Unfortunately, we are not aware of any work, in the literature,which addresses the cluster’s energy consumption and performance of various workloads in a simi-lar hybrid orchestration platform; when various approaches to scheduling and migration are takeninto account12.

11 https://ciao-project.github.io/

12 http://events17.linuxfoundation.org/sites/events/files/slides/Linuxcon%20NA%202016.pdf


Fig. 12: The CIAO architecture12

The HeporCloud framework uses the FillUp allocation policy to place user’s workloads onto avail-able resources in a workload-specific way. It means that most energy and performance efficient hostsare utilised first. However, the CIAO uses a first fit policy to place the workload onto resourceswithout any prioritisation with respect to energy and/or performance. Moreover, the HeporCloudframework selects the best candidate among various migration opportunities i.e. containers, VMsand/or bare-metal; using the consolidation with migration cost recovery (Cmcr) policy. Further-more, our orchestrator bias towards migrating containers first instead of VMs; as containers arelightweight that would finish their migration quicker than VMs. However, CIAO implements thesimple migration policy which selects the top most instance, that could be either a container, ora VM, in the migration list. Moreover, migration occurs when the utilisation of a node is lowerthan the lower utilisation threshold i.e. 20%. We assume that over utilisation of resources will nothappen; as the allocation and migration policies will not place instances or those nodes which doesnot have enough capacities. A number of studies [13] discuss the performance efficiency of bare-metal, VMs and containers, individually, but appear to ignore hybrid resource management andenergy efficiency, and with the notable exception of [38] this is rarely addressed. In [38], the authorssuggest that bare-metal could provide the highest performance at lowest energy cost. However, theevaluated results consider a single application (job); and, therefore, when a dynamic system isconsidered, then there results are not guaranteed to be optimal.Moreover, the authors in [36], [46] have discussed VMs, containers and containers runing insideVMs, but, bare-metal and energy efficiency are not explored. Sharma et al. [36] states that contain-ers in VMs provide performance benefit; and neighbouring containers within a VM can be trustedas well. Tay et al. [46] suggest exploring various technologies, such as VMs and containers, in thecontext of consolidation and workload migration. Felter et al. [32] studied resource managementof containers (Docker) and VMs (KVM), and compared the achievable performance of numer-ous applications with respect to bare-metal. Their evaluation suggests that VM’s and container’sperformance overlaps for certain kinds of workload. Moreover, the authors reject the idea that“IaaS should be implemented on VMs and PaaS on containers” – as there is no technical reason.


Unfortunately, migrations are not taken into account. Mondesire et al. [40] have investigated theperformance of bare-metal, VMs, containers and virtualised containers for executing interactivegame-based simulations, individually. The authors suggest that containers performance is compa-rable to the bare-metal; and mixing containers with VMs offer performance benefits between VMsand containers, alone. However, scheduling in a hybrid platform and consolidation of workloads interms of energy consumption are not investigated.Tchana et al. [47], [48] observed that VMS are, largely, underutilised in certain cloud datacenters;and proposed a solution called software consolidation. Software consolidation accommodate severalapplications, dynamically, on the same VM to minimise the number of used VMs. Moreover, soft-ware consolidation can be combined with VM consolidation that reduces the number of hosts in use.Software consolidation could reduce: (i) energy consumption of a private cloud; and (ii) the numberof used VMs, therefore, costs, in a public cloud. Their investigation suggests that approximately40% energy could be saved through software consolidation in their private cloud. Furthermore,approximately 40.5% user’s monetary cost could be saved in Amazon EC2 public cloud. This workclosely resembles our consolidation technique; however, their algorithm is very straightforward thatcould not decide effective migration among VMs, containers and software/applications in terms ofenergy consumption and workload performance.In the literature [4], [13], [36], [40], [48], [49], various migration techniques have been suggested forVMs and containers, but, individually. However, we are not aware of any work that investigatesthe impact of energy savings and workload performance degradation when migrations of VMs,containers, containers over VMs and bare-metal applications are taken into account, at the sametime. Similarly, no study describes scheduling for hybrid platforms in terms of using a centralisedscheduler and/or distributed schedulers for various resource platforms. Moreover, inter-platformand intra-platforms migrations, which are possible in hybrid datacenters, are also not discussedanywhere else. The summary of the comparison between our proposed HeporCloud and otherclosely related works is given in Table 18. We believe the information in Table 18 would also helpthe readers to quickly identify gaps for further research and investigation.

Related WorkParameters [4] [13] [32] [36] [40] [45] [46] [48] CIAO HeporCloud

VMs × × × × × × × × × ×Platforms Containers × × × × × × × × ×

Containers|VMs × × ×Bare-metal × × × ×

Energy consumption × × ×Metrics Workload performance × × × × × ×

User costs × × ×Scheduler Single ×

architecture Distributed × × × × × × × × ×Management Allocation × × × × × ×

policy Migration × × × ×

Table 18: Summary of the related work [the scheduler architecture applies to works in whichmultiple sandboxing technologies are being used; and the scheduler plus management policy areused to evaluate the platforms for various metrics]


8 Conclusions and Future Work

In this paper, we proposed a framework HeporCloud and, an integrated, workload-aware singleresource scheduler and orchestrator for hybrid cloud platforms. The proposed resource manager isable to allocate and predict effective workloads placement and migration decisions. Using reasonableassumptions, our empirical evaluation suggests that HeporCloud can schedule and consolidatevarious kinds of workloads energy, performance and, therefore, cost efficiently. Our investigationsuggests that: (i) using a, centralised, single scheduler is more energy and performance efficientthan using individual schedulers in hybrid platforms; (ii) under certain configurations, it might bemore energy and performance efficient not to migrate workloads; and (iii) inter-platform migrationsare more energy efficient than intra-platforms migrations, however, performance of the workloadvaries significantly. Moreover, containers are more energy, performance efficient than VMs; andenergy efficient than bare-metal hardware due to high level of resource utilization. Furthermore,for certain kinds of workloads, virtualised containers may be as bad as good, as compared to VMsand containers.In future, our objective would be to validate the proposed HeporCloud framework on a real cloudtest-bed; through importing it in the OpenStack [13]. OpenStack is a cloud management tool thatoperates and controls large pools of computational, storage, and network resources throughout adatacenter. Technically, the objective would be to propose an architecture or a resource manager tothe OpenStack community in order to unify the Ironic, Nova, Magnum and kolla services operatingraw hardware (bare-metal), virtual machine (VM), system containers and virtualised containers,respectively. Moreover, migrations could affect the workload performance, particularly, if a singleVM|container is migrated several times (repeatable migrations) [50]. In the future, we would adda control mechanism, to the proposed consolidation framework, in order to avoid such expensive,repeatable, migrations.

Acknowledgement

This work is supported by Abdul Wali Khan University, Mardan (AWKUM) and Higher EducationCommission (HEC), Pakistan.

References

1. A Shehabi, SJ Smith, N Horner, I Azevedo, R Brown, J Koomey, E Masanet, D Sartor, M Herrlin, andW Lintner. United states data center energy usage report. Lawrence Berkeley National Laboratory,Berkeley, California. LBNL-1005775 Page, 4, 2016.

2. Muhammad Zakarya. Energy, performance and cost efficient datacenters: A survey. Renewable andSustainable Energy Reviews, 94:363–385, 2018.

3. Muhammad Zakarya and Lee Gillam. Energy efficient computing, clusters, grids and clouds: A taxon-omy and survey. Sustainable Computing: Informatics and Systems, 14:13–33, 2017.

4. Muhammad Zakarya. An extended energy-aware cost recovery approach for virtual machine migration.IEEE Systems Journal, 2018.

5. Ohad Shai, Edi Shmueli, and Dror G Feitelson. Heuristics for resource matching in intel’s computefarm. In Workshop on Job Scheduling Strategies for Parallel Processing, pages 116–135. Springer, 2013.

6. Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bian-chini. Resource central: Understanding and predicting workloads for improved resource managementin large cloud platforms. In Proceedings of the 26th Symposium on Operating Systems Principles, pages153–167. ACM, 2017.

7. Charles Reiss, John Wilkes, and Joseph L Hellerstein. Google cluster-usage traces: format+ schema.Google Inc., Mountain View, CA, USA, Technical Report, 2011.

8. Muhammad Zakarya and Lee Gillam. Energy and performance aware resource management in hetero-geneous cloud datacenters. PhD thesis, University of Surrey, 2017.

9. Sareh Fotuhi Piraghaj, Amir Vahid Dastjerdi, Rodrigo N Calheiros, and Rajkumar Buyya. Container-cloudsim: An environment for modeling and simulation of containers in cloud data centers. Software:Practice and Experience, 47(4):505–521, 2017.


10. Rajkumar Buyya, Satish Narayana Srirama, Giuliano Casale, Rodrigo Calheiros, Yogesh Simmhan,Blesson Varghese, Erol Gelenbe, Bahman Javadi, Luis Miguel Vaquero, Marco AS Netto, et al. A man-ifesto for future generation cloud computing: research directions for the next decade. ACM computingsurveys (CSUR), 51(5):105, 2018.

11. H P technical white paper. Linux container performance on hpe proliant servers: Understandingperformance differences between containers and virtual machines, 2016.

12. Fei Xu, Fangming Liu, and Hai Jin. Heterogeneity and interference-aware virtual machine provisioningfor predictable performance in the cloud. IEEE Transactions on Computers, 65(8):2470–2483, 2016.

13. Charalampos Gavriil Kominos, Nicolas Seyvet, and Konstantinos Vandikas. Bare-metal, virtual ma-chines and containers in openstack. In Innovations in Clouds, Internet and Networks (ICIN), 201720th Conference on, pages 36–43. IEEE, 2017.

14. Ayaz Ali Khan, Muhammad Zakarya, and Rahim Khan. H2–a hybrid heterogeneity aware resourceorchestrator for cloud platforms. IEEE Systems Journal, 2019.

15. Tiago C. Ferreto, Marco A S Netto, Rodrigo N. Calheiros, and Cesar A F De Rose. Server consolidationwith migration control for virtualized data centers. Future Generation Computer Systems, 27(8):1027–1034, 2011.

16. Varun Gupta. Stochastic models and analysis for resource management in server farms. PhD thesis,Intel Corporation, 2011.

17. Anshul Gandhi, Varun Gupta, Mor Harchol-Balter, and Michael Kozuch. Energy-efficient dynamiccapacity provisioning in server farms. School of Computer Science, Carnegie Mellon University, Tech.Rep. CMU-CS-10-108, 2010.

18. Rodrigo N Calheiros, Rajiv Ranjan, Anton Beloglazov, Cesar AF De Rose, and Rajkumar Buyya.Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation ofresource provisioning algorithms. Software: Practice and Experience, 41(1):23–50, 2011.

19. George Amvrosiadis, Jun Woo Park, Gregory R Ganger, Garth A Gibson, Elisabeth Baseman, andNathan DeBardeleben. Bigger, longer, fewer: what do cluster jobs look like outside google? Technicalreport, Technical Report CMU-PDL-17-104, Carnegie Mellon Univedrsity Parallel Data . . . , 2017.

20. Alexey Tumanov, Angela Jiang, Jun Woo Park, Michael A Kozuch, and Gregory R Ganger. Jamaisvu:Robust scheduling with auto-estimated job runtimes. Technical report, Technical Report CMU-PDL-16-104. Carnegie Mellon University, 2016.

21. Warren Smith, Ian Foster, and Valerie Taylor. Predicting application run times with historical infor-mation. Journal of Parallel and Distributed Computing, 64(9):1007–1016, 2004.

22. Mehiar Dabbagh, Bechir Hamdaoui, Mohsen Guizani, and Ammar Rayes. Release-time aware vmplacement. In Globecom Workshops (GC Wkshps), 2014, pages 122–126. IEEE, 2014.

23. Charles Reiss, Alexey Tumanov, Gregory R. Ganger, Randy H. Katz, and Michael a. Kozuch. Het-erogeneity and dynamicity of clouds at scale. Proceedings of the Third ACM Symposium on CloudComputing - SoCC ’12, pages 1–13, 2012.

24. Muhammad Zakarya and Lee Gillam. Managing energy, performance and cost in large scale heteroge-neous datacenters using migrations. Future Generation Computer Systems, 93:529–547, 2019.

25. Muhammad Zakarya and Lee Gillam. An energy aware cost recovery approach for virtual machinemigration. In International Conference on the Economics of Grids, Clouds, Systems, and Services,pages 175–190. Springer, 2016.

26. Ibrahim Alzamil and Karim Djemame. Energy prediction for cloud workload patterns. In InternationalConference on the Economics of Grids, Clouds, Systems, and Services, pages 160–174. Springer, 2016.

27. Maxime Colmant, Mascha Kurpicz, Pascal Felber, Loıc Huertas, Romain Rouvoy, and Anita Sobe.Process-level power estimation in vm-based systems. In Proceedings of the Tenth European Conferenceon Computer Systems, page 14. ACM, 2015.

28. Mar Callau-Zori, Lavinia Samoila, Anne-Cecile Orgerie, and Guillaume Pierre. An experiment-drivenenergy consumption model for virtual machine management systems. Sustainable Computing: Infor-matics and Systems, 18:163–174, 2018.

29. Adrien Lebre, Jonathan Pastor, Anthony Simonet, and Mario Sudholt. Putting the next 500 vmplacement algorithms to the acid test: The infrastructure provider viewpoint. IEEE Transactions onParallel and Distributed Systems, 30(1):204–217, 2019.

30. Han-Peng Jiang and Wei-Mei Chen. Self-adaptive resource allocation for energy-aware virtual machineplacement in dynamic computing cloud. Journal of Network and Computer Applications, 120:119–129,2018.

31. Roberto Morabito, Jimmy Kjallman, and Miika Komu. Hypervisors vs. lightweight virtualization: aperformance comparison. In 2015 IEEE International Conference on Cloud Engineering, pages 386–393. IEEE, 2015.


32. Wes Felter, Alexandre Ferreira, Ram Rajamony, and Juan Rubio. An updated performance comparisonof virtual machines and linux containers. In Performance Analysis of Systems and Software (ISPASS),2015 IEEE International Symposium on, pages 171–172. IEEE, 2015.

33. John O’Loughlin and Lee Gillam. Performance evaluation for cost-efficient public infrastructure clouduse. In International Conference on Grid Economics and Business Models, pages 133–145. Springer,2014.

34. Bowen Ruan, Hang Huang, Song Wu, and Hai Jin. A performance study of containers in cloudenvironment. In Asia-Pacific Services Computing Conference, pages 343–356. Springer, 2016.

35. Vıctor Medel, Omer Rana, Jose Angel Banares, and Unai Arronategui. Modelling performance &resource management in kubernetes. In 2016 IEEE/ACM 9th International Conference on Utility andCloud Computing (UCC), pages 257–262. IEEE, 2016.

36. Prateek Sharma, Lucas Chaufournier, Prashant J Shenoy, and YC Tay. Containers and virtual ma-chines at scale: A comparative study. In Middleware, page 1, 2016.

37. Zhanibek Kozhirbayev and Richard O Sinnott. A performance comparison of container-based tech-nologies for the cloud. Future Generation Computer Systems, 68:175–182, 2017.

38. Sebastien Vaucher. Comparing virtual machines and linux containers, 2015.39. Mehiar Dabbagh, Bechir Hamdaoui, Mohsen Guizani, and Ammar Rayes. An energy-efficient vm pre-

diction and migration framework for overcommitted clouds. IEEE Transactions on Cloud Computing,2016.

40. Sean C Mondesire, Anastasia Angelopoulou, Shehan Sirigampola, and Brian Goldiez. Combining vir-tualization and containerization to support interactive games and simulations on the cloud. SimulationModelling Practice and Theory, 93:233–244, 2019.

41. John O’Loughlin. A workload-specific performance brokerage for infrastructure clouds. PhD thesis,University of Surrey, 2018.

42. Marcelo Amaral, Jorda Polo, David Carrera, Iqbal Mohomed, Merve Unuvar, and Malgorzata Steinder.Performance evaluation of microservices architectures using containers. In 2015 IEEE 14th Interna-tional Symposium on Network Computing and Applications, pages 27–34. IEEE, 2015.

43. Sai Venkat Naresh Kotikalapudi. Comparing live migration between linux containers and kernel virtualmachine: investigation study in terms of parameters, February 2017.

44. Pawe l Lubomski, Andrzej Kalinowski, and Henryk Krawczyk. Multi-level virtualization and its impacton system performance in cloud computing. In International Conference on Computer Networks, pages247–259. Springer, 2016.

45. Ilias Mavridis and Helen Karatza. Performance and overhead study of containers running on top ofvirtual machines. In 2017 IEEE 19th Conference on Business Informatics (CBI), volume 2, pages32–38. IEEE, 2017.

46. YC Tay, Kumar Gaurav, and Pavan Karkun. A performance comparison of containers and virtualmachines in workload migration context. In Distributed Computing Systems Workshops (ICDCSW),2017 IEEE 37th International Conference on, pages 61–66. IEEE, 2017.

47. Alain Tchana, Noel De Palma, Ibrahim Safieddine, Daniel Hagimont, Bruno Diot, and NicolasVuillerme. Software consolidation as an efficient energy and cost saving solution for a saas/paascloud model. In European Conference on Parallel Processing, pages 305–316. Springer, 2015.

48. Alain Tchana, Noel De Palma, Ibrahim Safieddine, and Daniel Hagimont. Software consolidation asan efficient energy and cost saving solution. Future Generation Computer Systems, 58:1–12, 2016.

49. Shripad Nadgowda, Sahil Suneja, Nilton Bila, and Canturk Isci. Voyager: Complete container statemigration. In Distributed Computing Systems (ICDCS), 2017 IEEE 37th International Conference on,pages 2137–2142. IEEE, 2017.

50. Ayaz Ali Khan, Muhammad Zakarya, and Rahim Khan. Energy-aware dynamic resource managementin elastic cloud datacenters. Simulation Modelling Practice and Theory, 92:82–99, 2019.

Date post:	12-Jun-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

HeporCloud: An energy, performance and cost e cient ... · run users’ applications in containers,...

Documents