Prototypical Implementation of aCloud Experimentation

Environment by an OpenStack FakeDriver

Stephan MannhartSteinach, Switzerland

Student ID: 11-917-515

Supervisor: Patrick G. Poullie, Dr. Thomas BocekDate of Submission: July 1, 2015

University of ZurichDepartment of Informatics (IFI)Binzmühlestrasse 14, CH-8050 Zürich, Switzerland ifi

VE

RT

IEF

UN

GS

AR

BE

IT–

Com

mun

icat

ion

Sys

tem

sG

roup

,Pro

f.D

r.B

urkh

ard

Stil

ler

VertiefungsarbeitCommunication Systems Group (CSG)Department of Informatics (IFI)University of ZurichBinzmühlestrasse 14, CH-8050 Zürich, SwitzerlandURL: http://www.csg.uzh.ch/

Contents

I Introduction 1

II Cloud Experimentation Environment 1

II.1 OpenStack Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

II.2 OpenStack Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

II.2.1 Identity Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

II.2.2 Image Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

II.2.3 Compute Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

II.2.4 Initial LXC Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

II.2.5 LXC Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

II.2.6 Networking Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

II.2.7 Fake Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

II.3 Nova Compute Command-line Client . . . . . . . . . . . . . . . . . . . . . . . . . 5

II.3.1 VM Management Commands . . . . . . . . . . . . . . . . . . . . . . . . . 6

II.3.2 VM Information Commands . . . . . . . . . . . . . . . . . . . . . . . . . . 6

II.3.3 Host Information Commands . . . . . . . . . . . . . . . . . . . . . . . . . 6

II.3.4 Diagnostic Data for Simulations . . . . . . . . . . . . . . . . . . . . . . . 6

IIIPerformance Limitations 7

III.1 Compute Host Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

III.2 VM Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

IVCRRA Implementation 8

IV.1 OpenStack Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

IV.1.1 Resource Consumption Data . . . . . . . . . . . . . . . . . . . . . . . . . 8

IV.1.2 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

IV.1.3 Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

IV.2 Integrating CRRA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

IV.2.1 Starting Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

IV.2.2 Conducting Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

IV.2.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1

Abstract

OpenStack is a powerful software toolkit to manage production clouds but when itcomes to simulating and experimenting with different cloud setups, it doesn’t providea ready to use solution. However, trying out different hypervisor and virtual machineratios as well as different resource consumptions can lead to interesting insights abouthow to increase the effectiveness of a cloud. This paper shows the installation andconfiguration of an OpenStack based experimentation environment with virtualisedcompute hosts and faked virtual machines.

Keywords: OpenStack, LXC, fake driver, cloud experimentation environment

I. Introduction

This paper is structured into three mainparts: First I will provide insight into thecloud experimentation environment thathas been set up to serve as an interfacefor the simulations. After that I will fo-cus on performance limitations in regardsto the used approach and finally, I will dis-cuss the chosen concept of implementing acloud runtime resource allocation (CRRA)simulation into the cloud experimentationenvironment.

II. Cloud ExperimentationEnvironment

The cloud experimentation environment(CEE) consists of the OpenStack “set ofsoftware tools for building and managingcloud computing platforms” [1], LXC con-tainers for the various OpenStack computenodes and the fake driver handling all VMoperations. Following, I discuss all men-tioned components of the environment inmore detail to provide a thorough overview.

II.1. OpenStack Architecture

LXC containerLXC containerLXC container

N15 physical host

nova computenova computenova compute

openstack controller & keystone

& glance

Figure 1: CEE OpenStack ConfigurationOverview

In Figure 1 you can see the general Open-Stack configuration that I chose for thephysical host N15 where I set up the CEE.OpenStack consists of a number of servicesthat need to be installed on different hosts.

horizon The dashboard that can be ac-cessed through a web browser to sim-plify OpenStack configuration tasksand provide an intuitive user interfaceto manage VMs.

nova The compute service that managescreating, scheduling and destroyingVMs.

neutron The new network service that wasintroduced with the Folsom releaseof OpenStack. This service shouldeventually replace the legacy ”nova-network” service [3]. This service

2 II CLOUD EXPERIMENTATION ENVIRONMENT

manages the network connections fordevices that are managed by otherOpenStack services, like VMs. Sincethe CEE does not use real VMs, thelegacy nova-network service suffices.

keystone The identity service that handlesauthentication and authorisation forall OpenStack services.

glance The image service that managesvirtual machine disk images to beused by the nova compute service.

The CEE discussed in this paper uses thelisted services. Additional unused servicesinclude swift for unstructured data manage-ment, cinder for block storage, ceilometerfor monitoring and metering, heat for cloudorchestration and trove for database-as-a-service functionality.

The main host is called “controller node” [2]and runs the OpenStack base-installation aswell as the keystone and glance services. Inorder to allow me to use multiple computenodes to host the hypervisor part of Open-Stack, I used LXC containers [4]. Each LXCcontainer runs its own nova compute serviceto serve as a hypervisor for the OpenStackcloud. The controller sees the LXC contain-ers as real compute hosts since the LXC vir-tualisation allows for a complete sandboxingof the compute host to simulate a real host.

II.2. OpenStack Installation

The whole installation was done for theJuno release of OpenStack with the help ofthe official OpenStack installation guide forUbuntu 14.04 [2]. The guide provides step-by-step instructions to set up an OpenStackinstallation with a controller node and onecompute node. The installation required 10individual passwords to set up proper au-thentication for all services. The most im-portant password is the keystone password

for the admin tenant. Tenants are “contain-ers used to group or isolate resources. Ten-ants also group or isolate identity objects”[5]. The admin tenant allows full access tothe nova compute service.

Initial setup of the OpenStack installationincluded preparing the SQL database whichis used by most OpenStack services to storetheir information and setting up the con-troller node as an NTP server for the com-pute nodes to synchronise time with. Mari-aDB was used as an SQL server as it wasrecommended by the installation guide.

II.2.1 Identity Service

The use of the identity service and its au-thentication can be simplified by using asmall bash script that can be sourced to setthe proper username, password and authen-tication URL for the identity service as en-vironment variables.

1 e xpo r t OS TENANT NAME=admin2 e xpo r t OS USERNAME=admin3 e xpo r t OS PASSWORD=

ADMIN PASSWORD4 e xpo r t OS AUTH URL=ht tp ://

c o n t r o l l e r :35357/ v2 . 0

Listing 1: Identity Settings

Listing 1 shows the bash script thatcan be sourced with $ source

identity_settings.sh to set up the au-thentication variables to use the commandline clients in OpenStack without needingto supply the authentication data as pa-rameters for every command.

II.2.2 Image Service

The image service provides an interface fornova compute to access the VM images itneeds. The authentication works throughkeystone and is set up permanently in the

II.2 OpenStack Installation 3

glance configuration file. In order to prop-erly use the glance image service, I createdan initial image using CirrOS [6] that I laterused as the main image to start new in-stances since all VMs will only be simulatedby the fake driver and the image will onlybe needed for issuing a valid instance-bootcommand.

II.2.3 Compute Service

To manage hypervisors for OpenStack com-pute purposes, the nova compute service isneeded. The service has to be installed intwo steps: First step is to install the com-pute service parts for the controller nodewhich consist of:

nova-api To handle user API-requests tothe compute service.

nova-cert To manage certificates.

nova-conductor To mitigate betweennova services and the underlying SQLdatabase.

nova-scheduler To schedule new VM in-stances to the correct compute host.

nova-consoleauth, nova-novncproxyTo allow authentication for consoleaccess like VNC to VM instances.

python-novaclient A command lineclient to directly interface with thenova API to execute nova computecommands. This client was heavilyused to build the CEE and to test theperformance properties of the CEE.

The second step is to install the nova com-pute services that are needed on the indi-vidual compute hosts.

II.2.4 Initial LXC Setup

The LXC-based compute hosts only neededthe setup of the nova-compute service to ac-cept VM instances that were scheduled bythe controller and to successfully managethese instances throughout their lifecycle.

Since the LXC compute hosts deviated fromnormal compute nodes in that they them-selves are virtualised, the initial installa-tion did not work as outlined in the in-stallation instructions [2]. After thoroughlyanalysing the nova-compute upstart log filesin /var/log/upstart/nova-compute [7], Irealised that the nova compute service triesto load the kernel module“nbd” for networkblock devices[8] in its upstart script locatedin /etc/init/nova-compute.conf . SinceLXC containers are not allowed to load ker-nel modules, the default upstart script failedand I needed to remove the kernel moduleloading. This presents no problem since noreal VMs are running on the compute hostand therefore network block devices are notneeded. It would however be possible to stilluse the nbd kernel module within an LXCcontainer by loading it on the LXC con-tainer’s host and since the LXC containershares the kernel with the host, the mod-ule would already be loaded when the LXCcontainer starts [9].

To simplify the IP address assignment forlarge amounts of LXC containers, the dns-masq DHCP server [10] was used to auto-matically assign IP addresses based on thehost name. For each host, an entry with theform dhcp-host=hostname,ip_address wasset up in the file /etc/lxc/dnsmasq.conf

on the controller node. This allows for aconvenient centralised IP management of alarge amount of LXC containers.

While testing the CEE, I ran into the prob-lem of dnsmasq leases that did not ex-pire fast enough and caused wrong IP ad-dress assignments for LXC containers. This

4 II CLOUD EXPERIMENTATION ENVIRONMENT

was mostly due to LXC containers hav-ing been removed and new ones being cre-ated with the same name for test purposes.The issued leases for dnsmasq are storedin /var/lib/misc/dnsmasq.lxcbr0.leases

. To remove old leases, they can just be re-moved from the leases file. To make thesechanges effective, the lxc-net service hasto be restarted with # service lxc-net

stop # service lxc-net start . It is im-portant to restart lxc-net with stop followedby start instead of restart because there is aknown bug that prevents restart from cor-rectly restarting dnsmasq [11]. The CEEin it’s final configuration does not exhibitthis problem since all hosts are pre-createdand only get started or stopped accordingto the needed amount of hosts, which doesnot create the discussed lease problem.

II.2.5 LXC Usage

I created the initial LXC container froma 64bit Ubuntu 14.04 image that was di-rectly downloaded through the lxc CLIwith $ sudo lxc-create -t download -n

nova_0 to create an LXC container withthe name nova 0. This container was thencustomised to work as a generic nova com-pute host template to be cloned to generatea multitude of containers.

1 #!/ b in / bash23 count=‘ l s / va r / l i b / l x c | t a i l

−1 | t r −d ”nova ” ‘4 new nova id=$ ( ( count + 1) ) ;5 new nova name=’nova ’

$new nova id ;67 l x c−c l o n e −o nova 0 −n

$new nova name89 # Add new dnsmasq r e s o l v e r

10 new nova ip end ing=$ ((100 +new nova id ) ) ;

11 echo ”dhcp−hos t=nova $new nova id , 1 0 . 0 . 3 .

$new nova ip end ing ” >> /e t c / l x c /dnsmasq . con f

12 s e r v i c e l x c−net r e s t a r t1314 # Change i p i n nova . con f15 sed − i ’ s / IP / 1 0 . 0 . 3 . ’

$new nova ip end ing ’ / g ’ /va r / l i b / l x c /$new nova name/ r o o t f s / e t c /nova/nova . con f

Listing 2: LXC Cloning Script

The bash script in Listing 2 simplifies thecloning of the nova 0 template by automat-ically creating an LXC container with incre-mented host name of the form nova numberand adding the necessary host entry to thednsmasq configuration file as well as settingup the correct IP address in the new hostsnova configuration.

14:21 n15 / $ sudo lxc-ls --fancyNAME STATE IPV4 IPV6 GROUPS AUTOSTART ----------------------------------------------------nova_0 STOPPED - - - NO nova_1 RUNNING 10.0.3.101 - - NO nova_2 STOPPED - - - NO nova_3 STOPPED - - - NO

Figure 2: LXC container list

To list the existing LXC containers, youcan use $ sudo lxc-ls --fancy which pro-duces an output similar to Figure 2 withthe states and IP addresses of all LXC con-tainers that are currently available on thesystem.

To start container nova 1, execute $ sudo

lxc-start -dn nova_1 to start the con-tainer as a daemon. To access containernova 1, use $ sudo lxc-attach -n nova_1

to open a shell connection as user root. Tofinally stop container nova 1, you can use$ sudo lxc-stop -n nova_1 .

II.2.6 Networking Component

As described in section II.1, the CEE usesnova-network as the networking component

II.3 Nova Compute Command-line Client 5

for the nova services. I primarily madethis choice because no real VMs are presentand therefore providing a fully featured net-working component like neutron would havebeen pointless. The nova-network service ishowever still needed as I discovered: Whenstarting new instances, the network state ofthe instance is requested from the networkservice. With no network service available,the request will time out and cause the VMto go into an ERROR state from which itcan neither recover nor be destroyed. Aworkaround for this problem was to directlydestroy the instance through the MySQLdatabase [12]. To simplify this process,I wrote a small SQL procedure that candirectly be called with an instance-UUID:destroy_instance(instance_uuid) . Thisprocedure could be used for different sce-narios where an instance can’t be recov-ered from an ERROR state but is no longerneeded for the outlined network problem asthe nova-network service solves this.

II.2.7 Fake Driver

The central part of the nova-compute ser-vice is the virtualisation driver to interfacewith the hypervisor. There are many com-pute drivers available that can be used [13].The default driver is the libvirt driver whichis also the driver that has been tested themost [14]. The libvirt driver supports a mul-titude of hypervisors [15] and would there-fore be a good choice as a basic driver. TheCEE is actually able to fully use the libvirtdriver to start real VMs inside the LXC-based compute host. This was very helpfulto observe realistic behaviour in regards toscheduling or diagnostic information.

In order to allow useful simulations inthe CEE, a fake driver needs to beused to bypass the actual hypervisorand just fake VM actions like creat-ing, starting, stopping or getting diag-

nostic information. To set up the com-pute hosts for use with the fake driver,compute_driver=fake.FakeDriver has tobe set in /etc/nova/nova-compute.conf .After restarting the nova-compute servicewith # service nova-compute restart ,the changes will take effect and the fakedriver will now handle all VMs.

To make sure that the fake driver won’tbe constrained by the actual resourcesthat are available on the physical ma-chine, the quotas for each tenant can beadjusted to allow unlimited amounts ofmemory, cpu cores and disk space to beused by VMs within the tenant [16]. Tochange the quotas, the admin tenant idhas to be requested with # keystone

tenant-list to use it to set the quo-tas: # nova quota-update --instances

-1 --cores -1 --ram -1 --fixed-ips -1

--floating-ips -1 tenant_id . SinceOpenStack is written in Python andis open source, the fake driver can bechanged to accommodate our needs. Thepython source for the fake driver is locatedin /usr/lib/python2.7/dist-packages/

nova/virt/fake.py . Since every LXC com-pute host will need this source, changesto the driver would need to be installedseparately on each LXC container. Tocircumvent this, I set up a shared folderon the physical machine that will bemounted on each LXC container [17]. Icopied all relevant Python source files to/opt/nova on host N15 and set the mountpoint in the shared LXC configuration file/usr/share/lxc/config/ubuntu.common.conf

which is read by all LXC containers.

II.3. Nova Compute Command-line Client

The compute command line client [18] canbe used to directly talk to the nova API.

6 II CLOUD EXPERIMENTATION ENVIRONMENT

There are some essential commands to man-age VMs and get diagnostic informationabout compute hosts and VMs that I willlist in the following sections.

II.3.1 VM Management Commands

Create VM To create a new VM, you firsthave to get the flavor-id through #

nova flavor-list and the image-id of the CirrOS image through# nova image-list . With bothids, you can then execute # nova

boot --flavor flavor_id --image

image_id vm_name

Start VM # nova start vm_name

Stop VM # nova stop vm_name

II.3.2 VM Information Commands

List all VMs # nova list

Show high-level info about a VM# nova show vm_name

Show VM diagnostics# nova diagnostics vm_name

Show VMs for host nova 1# nova hypervisor-servers nova_1

List diagnostics for all VMs on nova 1# nova host-describe nova_1

II.3.3 Host Information Commands

Show all OpenStack services# nova service-list

Show services of all hosts# nova host-list

Show resources of host nova 1# nova hypervisor-show nova_1

II.3.4 Diagnostic Data for Simula-tions

In order to perform useful simulations, thereneeds to be a way to obtain diagnosticinformation from the CEE. This can beachieved through the outlined command-line client commands from the last two sec-tions. I would like to highlight two of themost useful commands in regards to simu-lations. When simulating resource usage inthe CEE, the most interesting metrics in-clude: CPU time, memory usage, diskusage and network usage of the individ-ual VMs as well as the compute hosts them-selves.

11:15 n15 /root # nova hypervisor-show nova_1+-------------------------+------------+| Property | Value |+-------------------------+------------+| cpu_info | ? || current_workload | 0 || disk_available_least | 0 || free_disk_gb | 599840 || free_ram_mb | 15872 || host_ip | 10.0.3.101 || hypervisor_hostname | nova_1 || hypervisor_type | fake || hypervisor_version | 1000 || id | 1 || local_gb | 600000 || local_gb_used | 160 || memory_mb | 32768 || memory_mb_used | 16896 || running_vms | 1 || service_disabled_reason | - || service_host | nova_1 || service_id | 5 || state | up || status | enabled || vcpus | 12 || vcpus_used | 8 |+-------------------------+------------+

Figure 3: Hypervisor Stats

Figure 3 shows the output of the # nova

hypervisor-show nova_1 command to dis-play hypervisor information about the com-pute host nova 1. We can get detailed infor-mation about disk- as well as memory-usagefrom the host as well as a general overviewabout VCPU usage.

7

11:38 n15 /root # nova diagnostics vm_1+------------------+---------+| Property | Value |+------------------+---------+| cpu0_time | 5254 || memory | 524288 || memory-actual | 524288 || memory-rss | 98172 || vda_errors | -1 || vda_read | 262144 || vda_read_req | 112 || vda_write | 5778432 || vda_write_req | 488 || vnet1_rx | 2070139 || vnet1_rx_drop | 0 || vnet1_rx_errors | 0 || vnet1_rx_packets | 26701 || vnet1_tx | 140208 || vnet1_tx_drop | 0 || vnet1_tx_errors | 0 || vnet1_tx_packets | 662 |+------------------+---------+

Figure 4: VM Diagnostics

To get information about the VMs, Fig-ure 4 shows the output of the # nova

diagnostics vm_1 command to obtain di-agnostic information about VM “vm 1”.These information are particularly interest-ing as they show CPU time, memory usage,disk usage and network usage. The CPUtime can be used to calculate the currentCPU usage in percentage with the total cputime from /proc/stat [19].

III. PerformanceLimitations

An important aspect of the CEE is the abil-ity to simulate big loads with many hosts aswell as many VMs running. To test the per-formance of the CEE, I conducted a set oftests and gathered metrics about the phys-ical host to determine the maximum loadthat could be simulated.

III.1. Compute Host Limitations

The physical host limits the amount of LXCcontainers that can be created since everycontainer needs a certain amount of diskspace as well as memory to run.

nova 1 nova 2 nova 3 nova 4 nova 5213.6 249.9 249.9 201.3 202.1

Table 1: LXC Container Maximum MemoryUsage in MB

Table 1 shows the maximum memory us-age for the initial 5 LXC test-containers.To obtain these metrics, I needed to bindthe cgroup mounts to /sys/fs/cgroup

for all LXC containers [20]. Afterthat, I was able to obtain the maximumamount of memory allocated to each con-tainer during it’s lifetime by viewing thecontent of /sys/fs/cgroup/memory/lxc/

nova_*/memory.max_usage_in_bytes . I cal-culated an average memory usage of 223.3MB which means the physical host N15 witha total of 62 GB of memory would haveenough memory to run 273 LXC containersconsidering that N15 itself needs an addi-tional 3 GB for it’s own operating system.

The bottleneck in respect to the amount ofLXC containers that can be sustained onN15 is the amount of disk space that anLXC container needs. Since all containersare clones of container “nova 0” they sharethe same amount of disk space they needto work. All files for container “nova 0”are stored in /var/lib/lxc/nova_0/rootfs

which amounts to 994 MB. Since N15 onlyhas 67.3 GB of usable disk space, it couldonly run a maximum of 69 LXC-based com-pute hosts.

8 IV CRRA IMPLEMENTATION

III.2. VM Limitations

0

1

2

3

4

5

6

7

8

9

0

10

20

30

40

50

60

70

80

90

100

00:0

101

:09

02:1

703

:25

04:3

305

:41

06:4

907

:57

09:0

510

:13

11:2

112

:29

13:3

714

:45

15:5

317

:01

18:0

919

:17

20:2

521

:33

22:4

123

:49

24:5

726

:05

27:1

328

:21

29:2

930

:37

31:4

532

:53

34:0

135

:09

36:1

737

:25

38:3

339

:41

40:4

941

:57

43:0

544

:13

45:2

146

:29

47:3

748

:45

mem

ory

usag

e (G

B)

cpu

usag

e (%

)

time (s)

cpu usagememory usage

Figure 5: Memory and CPU Usage for 1000VMs

In regards to VMs, the CEE turned outto be much more capable. In an attemptto stress the physical host, I started 1000VMs with one compute node (nova 1) run-ning. I used dstat to log the CPU andmemory usage [21] that resulted in Figure5 which clearly shows that the amount ofmemory needed to support the VMs ini-tially increases but starts to plateau aroundthe 30 minute mark. The memory usagenever exceeds 8 GB and it has to be notedthat the test was run on a freshly bootedmachine.

The behaviour in respect to CPU usageis also to be expected since the VMs arerunning on a faked hypervisor and there-fore consume no CPU time. The individualspikes every 10 minutes could be caused byother background processes and do not havesuch a huge significance as they only last fora single second.

IV. CRRA Implementation

The cloud runtime resource allocation orCRRA is an important addition to the ex-isting CEE that allows for simulated work-loads to be assigned to the VMs. Sincethe VMs do not actually exist and are only

faked by the fake hypervisor, CRRA be-comes an important tool to conduct simula-tions within the CEE. The nova command-line client that was discussed in section II.3allows detailed information-gathering withrespect to compute hosts as well as VMsbut in order to understand where this diag-nostic data comes from, we need to take alook at how scheduling in OpenStack works.

IV.1. OpenStack Scheduling

FiltersRetryFilterAvailabilityZoneFilterRamFilterComputeFilterComputeCapabilityFilterImagePropertiesFilter

WeightingRAM weightMetrics weightI/O Ops weight

host 1host 2host 3host 4

host 1host 2

host 4

host 2host 1host 4

Figure 6: OpenStack Scheduling

As Figure 6 shows, scheduling takes threesteps in OpenStack: First, all hosts are fil-tered by a predefined set of filters, then theyare weighted and finally, a host is chosenfrom the resulting list of potential hosts torun the VM [22].

IV.1.1 Resource Consumption Data

In order to apply the filters and weightsthat will be described in the follow-ing two sections, the nova schedulerneeds resource consumption data of allhosts. This data is directly accessedfrom the database by the scheduler andstored there by each host in a defaulttime interval of 60 seconds as definedin /usr/lib/python2.7/dist-packages/

nova/openstack/common/periodic_task.py

[27].

IV.1 OpenStack Scheduling 9

IV.1.2 Filtering

The first step in scheduling is filtering thecomplete list of all nova compute hosts.The set of filters to be used is defined in/etc/nova/nova.conf [23]. The default setof filters includes:

RetryFilter Skips nodes that have alreadybeen attempted for scheduling.

AvailabilityZoneFilter Makes sure theVM gets started on a node that is inthe availability zone as described inthe VM instance properties. Avail-ability zones are nodes on a singlecloud site that are somehow isolatedfrom the other nodes, for example bydistinct power supplies [24].

RamFilter Chooses nodes that haveenough free RAM. Since the ratioof physical to virtual RAM isn’t al-ways 1:1, the default nova.conf settingram_allocation_ratio = 1.5 is usedto allow overcommitting of RAM. Forexample, if a compute node only has1 GB RAM, it is still able to run VMsthat require 1.5 GB. This makes sensesince the allocation does not directlyresult in the complete consumption ofthe allocated RAM. If the VM that isset to require 1.5 GB RAM actuallyonly uses a maximum of 1 GB RAM,then the host it runs on can easilysupport it [23].

ComputeFilter Filters nodes that are ei-ther disabled or that are offline.

ComputeCapabilitiesFilter Checks ifnodes satisfy additional capabilitiesthat could be needed by the in-stance. Extra capabilities can also beset through the flavor as key:value

pairs.

ImagePropertiesFilter Filters nodesthat don’t satisfy the architecture, hy-pervisor type or virtual machine modethat was set for the VM’s image.

IV.1.3 Weighting

After the initial host list has been runthrough all filters, the list is weighted ac-cording to the following ruleset: Threeweighers are applied to the host list. Af-ter a weight has been assigned to eachhost, the weights are normalised by scal-ing each weight to the range between themaximum and minimum weight recorded:weight−minimum

range. Each normalised weight-

value is then multiplied by the definedweight multiplier. Following three weighersare available:

RAMWeigher The more free RAM, thehigher the weight. The weightmultiplier is set in nova.conf asram_weight_multiplier and defaultsto 1.0 [23].

MetricsWeigher Different metrics of thecompute host can be weighed ac-cording to defined weighting settingsthat are configured through nova.confas weight_setting with the formmetricName=ratio . As an exam-ple, if hosts with low previous load(low cpu utilisation) are wanted, aweight setting of the following formcould be used: weight_setting =

cpu.percent=-1.0" [25].

IoOpsWeigher The amount of input andoutput operations specify the weightfor the host. More I/O opera-tions give the host a higher weight.The default weight multiplier is -1.0, setting a preference on lightworkload compute hosts. To set apreference on heavy workload com-pute hosts, a positive multiplier has

10 IV CRRA IMPLEMENTATION

to be chosen. The weight mul-tiplier can be set in nova.conf asio_ops_weight_multiplier [26].

From the weighted list, a potential hostis chosen randomly. The choice is how-ever influenced by the nova.conf variablescheduler_host_subset_size . If this vari-able is bigger than the size of the hostlist, the final host will be randomly chosenamong all possible hosts from the list. Ifthe variable is smaller than 1, it will be setto 1 which results in the first host from thelist to be chosen as the final host. Everyother value in between makes the schedulerchoose a host randomly from a subset of theweighted host list.

IV.2. Integrating CRRA

The main problem concerning the CRRAintegration into the CEE was the communi-cation between the CRRA cloudlet and thenumerous LXC-based nova compute hosts.The CRRA cloudlet itself is written in Java[28] but the fake driver it needs to communi-cate with is written in Python. To solve thecommunication problem, I looked into dif-ferent approaches for interprocess communi-cation. There are generally two good waysto do interprocess communication on a unixbased system: Either through named pipesor through sockets [29]. After trying outnamed pipes with a small Python to Pythonprototype that I developed, I realised thatit is not very well suited for fast messageexchanges [30] and there is also the prob-lem of named pipes that are missing, havealready been created or have the wrong per-missions set. I finally decided to use TCPsockets for the interprocess communication[31]. The main advantage of this approachis the way I can run the CRRA cloudlet asa single instance on the physical host with-out needing to create multiple cloudlets for

each LXC container. This is based on thefact that each LXC container has it’s own IPand can therefore be directly contacted overTCP. Centralising the CRRA component ofthe CEE eliminates the need of a time syn-chronisation between simulations as they allrun within the same simulation instance.

N15 physical host

LXC-basednova compute host

LXC-basednova compute host

CRRACloudlet send resource

information inJSON over TCP

send resourceinformation inJSON over TCP

Figure 7: CRRA Cloudlet in the CEE

Figure 7 shows the overall architecture forthe communication between CRRA cloudletand OpenStack. The CRRA cloudlet di-rectly opens a TCP socket to each LXC hostand sends it updated resource informationevery second about each VM that is cur-rently running on the compute host. Thecompute host itself only listens for incomingTCP connections as soon as the fake driveris initialised.

The communication takes place in the portrange from 50001 to 50050 with 50 LXC-based compute hosts to avoid collisions withexisting TCP services [32]. The resource in-formation is exchanged in JSON because itis a well documented object notation withimplementations for Java as well as Python[33] [34].

IV.2.1 Starting Simulations

At the beginning of a simulation, the CRRAcloudlet starts the needed amount of LXCcontainers to serve as nova compute hosts

11

through the LXC command line client withlxc-start -dn nova_* . After the requiredamount of LXC hosts have been started, thecloudlet informs the LXC hosts through theTCP socket about their resource configura-tions in order to make sure the schedulingwill work realistically based on the resource-limitations as defined by the CRRA cloudletinput. To start a VM instance, the cloudletcan directly communicate with the novaAPI through curl [35].

The cloudlet then sends a request to all LXCcontainers through the TCP socket to ob-tain a list of VMs that have been sched-uled to run on the hosts by the nova sched-uler. With this information, the cloudletis now able to start simulating the resourceconsumptions of all VMs in real time andsend the resulting changes in resources tothe LXC hosts in JSON over the TCPsocket. The cloudlet needs to simulate inreal time because the resource consump-tions need to be requested directly fromthe compute hosts through the nova APIor nova command-line client. Faster thanreal-time simulations would result in gapsin the resource consumptions that are com-municated to the compute hosts and wouldpresent a lacking overall picture in regardsto the resource changes.

IV.2.2 Conducting Experiments

With the approach outlined in the last sec-tion, a researcher would now be able to di-rectly conduct an experiment in the CEE.The input parameters are the same as for aCloudsim simulation as the CRRA cloudletdirectly starts the compute hosts and VMsaccording to it’s input parameters.

The researcher would however need to setup a capturing environment where the out-put from the nova command-line client or

nova API in regards to resource consump-tions of VMs and compute hosts would needto be recorded in order to use the data forfurther analysis.

IV.2.3 Limitations

The outlined approach is mostly in con-cept stage as there is not yet a modifiedCRRA cloudlet available that would be ableto communicate with the compute hosts dueto time constraints. The fake driver mod-ifications are also still missing but experi-ments in regards to the TCP socket commu-nications have been made that allowed meto output random resource changes throughthe fake driver.

V. Conclusions

OpenStack is very powerful to manage andrun complex cloud configurations. The ba-sic OpenStack documentation was initiallyvery helpful in overcoming hurdles that pre-sented themselves during the OpenStack in-stallation process I went through. The spe-cific requirement of a fake hypervisor driverand nova compute hosts running inside LXCcontainers however made the process muchmore complicated since there are little re-sources available online in regards to thefake driver as well as running compute hostsinside LXC containers. Based on these con-straints, the CEE setup turned out to bemuch more complicated than initially ex-pected. Due to this fact, the implemen-tation of cloud runtime resource allocationcould not be completed but a prototypicalimplementation concept was developed thatis based on individual prototype tests aswell as discussions with the CRRA cloudletcreator Patrick Taddei, to ensure a soundconcept.

12 REFERENCES

Glossary

Cloudlet A class to model cloud-based application services in Cloudsim [36].

Folsom, Juno Release names for different OpenStack versions.

Hypervisor A software or hardware component to create and run virtual machines.

Mounting To make a storage device accessible through the file system.

N15 The physical host in the CSG testbed that runs the CEE.

Tenant Containers used to group or isolate resources. [5]

Virtual Machine An emulation of a computer system.

Abbreviations

API Application Programming InterfaceCPU Central Processing UnitCRRA Cloud Runtime Resource AllocationLXC Linux ContainersNBD Network Block DevicesRAM Random-Access MemoryVCPU Virtual CPUVM Virtual Machine

References

[1] Opensource.com: What is OpenStack?, http://opensource.com/resources/

what-is-openstack, (visited on 06/27/2015)

[2] OpenStack Foundation: OpenStack Installation Guide for Ubuntu 14.04, http:

//docs.openstack.org/juno/install-guide/install/apt/content/, (visited on06/27/2015)

[3] OpenStack Foundation: Deprecation of the Nova Network Service, http://docs.

openstack.org/openstack-ops/content/nova-network-deprecation.html, (vis-ited on 06/27/2015)

[4] Ubuntu Documentation Team: Ubuntu Virtualisation Server Guide, https://help.ubuntu.com/lts/serverguide/lxc.html, (visited on 06/27/2015)

[5] OpenStack Foundation: OpenStack Identity Concepts, http://docs.openstack.

org/juno/install-guide/install/apt/content/keystone-concepts.html, (vis-ited on 06/27/2015)

REFERENCES 13

[6] Canonical Ltd.: CirrOS, https://launchpad.net/cirros, (visited on 06/27/2015)

[7] Ask Openstack: Nova-scheduler Driver Setting Instance toError State, https://ask.openstack.org/en/question/1279/

nova-scheduler-driver-setting-instance-to-error-state/?answer=1408#

post-id-1408, (visited on 06/29/2015)

[8] Wikipedia: Network Block Device, https://en.wikipedia.org/wiki/Network_

block_device, (visited on 06/29/2015)

[9] Michael Foord: Workaround for modprobe inside LXC, https://bugs.launchpad.net/juju-core/+bug/1353443/comments/3, (visited on 06/29/2015)

[10] OpenStack Foundation: Dnsmasq DHCP Server, http://docs.openstack.org/

admin-guide-cloud/content/section_dnsmasq.html, (visited on 06/27/2015)

[11] James Page: Workaround for lxc-net Restart Bug, https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1043588/comments/2, (visited on 06/29/2015)

[12] Remy van Elst: Fix inconsistent Openstack Volumes and Instances from Cin-der and Nova via the Database, https://raymii.org/s/articles/Fix_

inconsistent_Openstack_volumes_and_instances_from_Cinder_and_Nova_

via_the_database.html, (visited on 06/29/2015)

[13] OpenStack Foundation: Nova.conf Configuration Options,http://docs.openstack.org/juno/config-reference/content/

list-of-compute-config-options.html#config_table_nova_compute, (vis-ited on 06/29/2015)

[14] OpenStack Foundation: HyperVisorSupportMatrix, https://wiki.openstack.

org/wiki/HypervisorSupportMatrix#Compute_Drivers, (visited on 06/29/2015)

[15] RedHat: libvirt.org, http://libvirt.org, (visited on 06/29/2015)

[16] Russell Bryant: OpenStack API Mocker or Simulator, https://ask.openstack.

org/en/question/28/openstack-api-mocker-or-simulator/?answer=52#

post-id-52, (visited on 06/29/2015)

[17] OpenSuSE: LXC Mount Shared Directory, https://en.opensuse.org/User:Tsu2/LXC_mount_shared_directory, (visited on 06/29/2015)

[18] OpenStack Foundation: Compute Command-Line Client, http://docs.

openstack.org/cli-reference/content/novaclient_commands.html, (visitedon 06/29/2015)

[19] LinuxHowTos.org: /proc/stat explained, http://www.linuxhowtos.org/System/

procstat.htm, (visited on 06/30/2015)

[20] Fabio Kung: Memory Inside Linux Containers, http://fabiokung.com/2014/03/13/memory-inside-linux-containers/#comment-3590, (visited on 06/30/2015)

14 REFERENCES

[21] Russell Barnes: Create a Graph of your System’s Performance, http://www.

linuxuser.co.uk/tutorials/create-a-graph-of-your-systems-performance/

4, (visited on 06/30/2015)

[22] OpenStack Foundation: Filter Scheduler, http://docs.openstack.org/

developer/nova/filter_scheduler.html?highlight=scheduler, (visited on06/30/2015)

[23] OpenStack Foundation: OpenStack Scheduling Configuration Reference,http://docs.openstack.org/juno/config-reference/content/section_

compute-scheduler.html, (visited on 06/30/2015)

[24] OpenStack Foundation: Scaling, http://docs.openstack.org/openstack-ops/

content/scaling.html#az_s3, (visited on 06/30/2015)

[25] Lianhao Lu, Yingxin Chen: Utilization-based Scheduling in OpenStack* Compute(Nova), https://01.org/sites/default/files/utilization_based_scheduing_in_openstack_compute_nova_1.docx, (visited on 06/30/2015)

[26] OpenStack Foundation: Create Nova Scheduler IO Ops Weighter, http:

//specs.openstack.org/openstack/nova-specs/specs/juno/approved/

io-ops-weight.html, (visited on 06/30/2015)

[27] Donald D. Dugger: Compute Node Update Interval, http://lists.openstack.org/pipermail/openstack/2014-July/008216.html, (visited on 06/30/2015)

[28] Patrick Taddei: Cloudsim-RDA on GitHub, https://github.com/pattad/

cloudsim-rda/tree/master/src/main/java/ch/uzh/ifi/csg/cloudsim/rda,(visited on 06/30/2015)

[29] Stackoverflow: Fastest Method for IPC between Java andC/C++, http://stackoverflow.com/questions/2635272/

fastest-low-latency-method-for-inter-process-communication-between-java-and-c,(visited on 06/30/2015)

[30] Adam Fraser: IPC between Python and Java, http://stackoverflow.com/a/

3804547, (visited on 06/30/2015)

[31] How to connect a Python Client to Java Server withTCP Socket, https://norwied.wordpress.com/2012/04/17/

how-to-connect-a-python-client-to-java-server-with-tcp-sockets/,(visited on 06/30/2015)

[32] Wikipedia: List of TCP and UDP Port Numbers, https://en.wikipedia.org/

wiki/List_of_TCP_and_UDP_port_numbers, (visited on 06/30/2015)

[33] Veer Shrivastav: Decoding JSON String in Java, http://stackoverflow.com/a/

16586100, (visited on 06/30/2015)

[34] Python Software Foundation: JSON Encoder and Decoder, https://docs.python.org/2/library/json.html, (visited on 06/30/2015)

REFERENCES 15

[35] Damion Parry: Create an OpenStack Instance with just Curl, https://catn.

com/2013/04/23/create-an-openstack-instance-with-just-curl/, (visited on06/30/2015)

[36] John G. Michopoulos: What is a Cloudlet in Cloudsim?, http://www.

researchgate.net/post/What_is_a_Cloudlet_in_Cloudsim, (visited on06/30/2015)