Research ArticleA Load Balancing Scheme Using Federate Migration Based onVirtual Machines for Cloud Simulations
Xiao Song Yaofei Ma and Da Teng
Science and Technology on Aircraft Control Laboratory School of Automation Science Beihang University Beijing 100191 China
Correspondence should be addressed to Yaofei Ma mayaofeibuaa163com
Received 4 June 2014 Revised 18 September 2014 Accepted 20 September 2014
Academic Editor Minrui Fei
Copyright copy 2015 Xiao Song et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited
A maturing and promising technology Cloud computing can benefit large-scale simulations by providing on-demand anywheresimulation services to users In order to enable multitask and multiuser simulation systems with Cloud computing Cloudsimulation platform (CSP) was proposed and developed To use key techniques of Cloud computing such as virtualization topromote the running efficiency of large-scale military HLA systems this paper proposes a new type of federate container virtualmachine (VM) and its dynamic migration algorithm considering both computation and communication cost Experiments showthat the migration scheme effectively improves the running efficiency of HLA system when the distributed system is not saturated
1 Introduction
Nowadays Cloud computing has been recognized as a newrevolution in the IT industry Key techniques in Cloud com-puting such as virtualization technology automatic deploy-ment resourcemanagementWeb services SOA high perfor-mance IO and high-speed internet are well-developed andhave been widely applied in various domains Accordinglyin the area of modeling and simulation (MampS) Cloud-basedcomputer simulation (CSim) [1ndash6] has been proposed anddesigned to provide users with better solutions to solve MampSproblems including large capital outlays in hardware but lowutilization high complexity in building simulation systemand high labor cost in simulation software maintenance
Currently CSim research is at its preliminary stage andthe pioneering works can be divided into three categories
(i) CSim framework and how to plant existing simulationsoftware into the cloud Liu et al [1] presents theprocess of deploying existing Parallel Discrete-EventSimulation (PDES) engine into the cloud the deploy-ment includes adjusting the structure of the modelto fit the features of Cloud computing developingthe simulation executionmode adding the horizontalscalability module to achieve service scalability andusing the resource allocator Li et al [2] proposed
Cloud simulation platform and its prototype archi-tecture which implement new techniques includingHLARTI (runtime infrastructure software imple-mentation ofHLA) technique based onWeb resourcedynamic management middleware technique basedon virtualization technique
(ii) Optimistic time advancement algorithms in the cloudJafer et al [3] presents the state of the art in PDES andit summarizes current research of PDES in the cloudsand hardware acceleration In order to address con-cerns about interference and communication delaysthat are inherent in Cloud computing environmentsFujimoto et al [4 5] consecutively studied paralleland distributed simulation in the cloud focusing onoptimistic parallel simulation advancement approachnamed ldquotime warp straggler message identificationprotocol (TW-SMIP)rdquo
(iii) Cloud agents and web-based simulations Javor andFur [6] proposes to solve high complexity problemsusing advanced Web services and Cloud computingtechniques It designs a framework in which cloudagents are composed of Web services integratingcomplex agent elements including large databases anddifferent novel concepts of inference engines
Hindawi Publishing CorporationMathematical Problems in EngineeringVolume 2015 Article ID 506432 11 pageshttpdxdoiorg1011552015506432
2 Mathematical Problems in Engineering
Meanwhile one of the most important MampS standardshigh level architecture (HLA) is an essential IEEE simu-lation interoperability standard in the area of distributedand parallel simulations And IEEE has developed HLAimplementation standards such as HLA-evolved [7ndash9] WithHLA and its implementation program runtime infrastructure(RTI) a lot of HLA compatible simulation systems have beendeveloped including large-scale military HLA applicationssuch as ModSAF [10] CCTT SAF [11] OneSAF [12 13]and COSIM [14] These large-scale simulations are usuallyhardware-consuming and software-intensive as they containnumerous models of environments and entities Grid tech-nology [15ndash17] has been used to solve the resource allocationand scheduling problem but Grid is not fully used whilecloud is more supported by many international companiessuch as Amazon [18] Google [19] and Softlayer [20]
As such we propose to use Cloud computing techniquesto develop simulation systems to solve the high cost andoverlapped hardware and software resources problem withcloud Cloud computing infrastructure offers numerous ben-efits such as reconfigurable dynamic resources and unifiedand simplified access to resources
Although the works in [1ndash6] explored new paradigmsof CSim few papers addressed the specific steps of how torunHLA simulations in cloud especially virtualization basedlarge-scale military applications Therefore in this paperwe mainly study how to run existing large-scale HLARTIsimulations with cloud and how to run themmore efficiently
The paper is structured as follows In Section 2 keytechnology of Cloud computing virtualization and HLAload balancing strategies are introduced and analyzed Corepart of the contribution Section 3 introduces the proposedvirtualization based HLA federation framework and run-time dynamicmigration algorithm Section 4 shows designedvalidation experiments and discusses results Finally conclu-sions are drawn and future works are discussed
2 Related Work
In this section virtualization related technologies and loadbalancing approaches are studied and analyzed
21 Virtualization Technology and HLA Simulations Incloud Virtualization technology (VT) encapsulates the sim-ulation application models and software packages to a rela-tively independent execution environment VM and deploysVMs to the underlying Physical Machines (PM or hosts) [21]Furthermore by the segmentation in space and the time-sharing in time [22 23] VT encapsulates the distribution andheterogeneity of the hardware resources allowing multiplevirtual machines to run simultaneously on one host as wellas the migration of VMs among different hosts In this wayVT greatly improves the resource utilization and quality ofservice and is gradually implemented in the medium-scaleand large-scale distributed simulation platform
One of the key virtualization technologies is VMmonitor(VMM) which connects applications and VMs with under-lying hardware to control the running of multiple operatingsystems VMM decouples the software from the hardware by
inserting a layer of interaction between the software runningin the VM and hardware so that VMM could obtain directcontrol of how operating systems in VMs access underlyinghardware resources [24] The design of VMM also makesit possible to implement the functions like live migration[25] and security that are difficult to achieve in modernoperating systemsThrough VMM computing resources canbe encapsulated into a collection of VMs according to userrsquosrequirements and VMs can be remapped and replicatedeasily In addition the state of VMs can be suspendedand resumed at any time and the running context can berolled back to a checkpoint Thus virtualization technologyprovides the ability to consolidatemanagement of distributedand heterogeneous resources by means of decoupling soft-ware and hardware transparent access encapsulation livemigration and resources isolation
Compared with the federate migration method of [1726] the main difference is that we use Virtual Machines(VMs) to be containers of federates From the perspective ofpractical implementation using cloud-based VMs to handlethe resource management and dynamic migration of HLAsimulation systems has the following merits compared withprevious approaches
(i) It saves programming efforts to implement migrationof federates with the support of the virtualizationmiddleware interfaces developed by many businessor open-source VM vendors such as VMware [27]Opennebula [28] Eucalyptus [29] and Xen [30]
(ii) VM live (or online) migration technique makes itfeasible to realize high efficient federate migrationPrevious works show that VM live migration down-time is mainly proportional to the amount of theldquowritable working setrdquo and could have downtime aslow as 60 to 200ms [31 32]
(iii) VM encapsulating HLA application software pack-ages such as RTI could be copied easily This greatlysaves the time and efforts simulation practitionersspend to configure the operational system and soft-ware environment which is often a tricky and tediousprocess especially during the large-scale simulationapplications deployment phase
Therefore it is more convenient for simulationists todevelop and run large-scale simulations if we adapt thevirtualization technology That is why we study cloud-basedHLA simulation in this paper However when we are ensuredthat we could use VMs to help us handle the federatemigrations we also encounter the problems that using VM toencapsulate federates create additional overhead Someworks[1 33 34] have designed experiments to find performanceloss caused by virtualizationThey found that the applications(with different event interaction complexity) ran in VM havethe maximum loss of 333 compared to ran in PhysicalMachine Although the performance loss is not high westill have to take more measures to make HLA federationrun faster One effective way is to improve current loadbalancing algorithms with better heuristics which considersHLA communication cost among federates
Mathematical Problems in Engineering 3
22 Load Balance Strategy in HLA and Beyond A famousdeveloping IEEE standard in Modeling and Simulation(MampS) society HLA is also gradually known for its slowrunning characteristics One important reason is that HLAis short of load-balance methods [17 26 35ndash38] to make theheavy load host run faster while all the other nodes must waituntil the slowest federate advances so they can move to thenext safe lookahead
Current studies of HLA load balance techniques could bedivided into two categories
(i) Load Balance Algorithms for Distributed and Parallel Sys-tems Including HLA Load balance strategies for distributedand parallel systems are basically adaptable for HLA whichis designed to provide a common set of framework andrules for distributed simulation In load balance algorithmsthe core concerns are computing and communication costs[17 26 39ndash43] Our algorithm has been influenced by earlierworks in three main areas heuristics computation andcommunication cost
Most dynamic load balancing works use heuristic algo-rithms because of its NP-hard nature [39] Paper [40] studiedheuristic methods for dynamic load balancing in a message-passing supercomputer It uses easy-to-implement heuristics(choosing a node with minimum load as the migration desti-nation node) and variable threshold in migrating processesamong the multicomputer nodes Genetic algorithms areused in [41] to dynamically distribute simulation tasks andcompared with a first fit algorithm and a random allocationscheme In this work the test results show that geneticmethod outperforms the other two both in completion timeand average processor utilization In paper [39] adaptiveload threshold is emphasized to suit the changing load onthe system and its basic strategy is also to ensure thatheavily loaded node is balanced first with lightly loaded nodeSimulated annealing algorithm is addressed in paper [42]which transforms the problem into a NP-complete problemnamed degree-constrained minimum spanning tree Thisapproach needs to adjust the heuristic factors according tothe practical applications These works use heuristics as anessential approach which is also our basic strategy becauseit is simple practical and time-saving especially in large-scale distributed simulation However most of the traditionalworks are one-by-one migration while in our approach it isdesigned to migrate a set of interaction-aware VMs to savethe cost of migration procedure latency and communication
Moreover papers [17 26] study dynamic load balancingusing Grid services for HLA-based large-scale simulationsThese pioneering works store all the resources in a queue andorganize them in ascendant order and themigration pairs areassembled with both extremities The algorithm is practicaland simple but it does not consider the communication costFujimoto et al in [4 5] addressed ldquoNetwork traffic and com-munication delays are significant in current implementationof Cloud computing infrastructurerdquo Thus for the cloud-based large-scale simulations communication cost must betaken into account in dynamic HLA load balancing
All the abovementioned works are meaningful but HLAsimulation systems have some more distinct characteristics
which are neglected by these researches For instance allinteractions among HLA federates are mainly modeled andimplemented with interaction classes and object classes Inmost cases of military simulations the former classes arestochastic and the latter classes are periodic For example thestatus data describing the position of a tank in Cartesiancoordinates would be modeled as an object class and itsinstance data is updated by the tank entity at every timestep The data describing the fire of a tank entity would bemodeled as an interaction class and its instance data is sentwhen the tank fires Moreover the interaction classes objectclasses and their relations of subscription and publicationare all defined before the simulation starts Therefore despitethe stochastic interaction class communications we can stillcompute the communication cost of periodic object classesand these core HLA characteristics will be considered in theload balance strategy of this paper
(ii) Federate Migration Techniques The approach presentedby Tan et al [35 36] accomplishes federate migration witha federate wrapper which controls the federates executionthrough SugarCubes stopping and resuming a federate Thekey point of this research is using the third-partymechanismsto avoid HLA federation saverestore approach Althoughit gains less latency than some other works its minimumlatency is about 8 seconds which is much longer than VMapproach in this paper
To propose and implement an efficient procedure forbalancing HLA simulationrsquos load paper [17] migrates feder-ates through the GRAM Gridsrsquo service such as Web servicesgrid resource allocation and management (WS GRAM)and GridFTP To make the migration delay as negligibleas possible the authors implement the migration in twosteps In the first step it executes in a way that federatedoes not stop its execution while initialization files aretransferred In the second step the RTI methods are calledto freeze the federation and the rest of data regarding thefederatersquos running status and messages are transferred Thetwo migration steps are well-devised and similar strategy isimplemented in this paper while using live VMmigration
Furthermore in the companion work of [17] thesis [26]decreases the migration latency to around 08 seconds whichis highly efficiency However as we described in the previoussection VM live migration downtime is mainly proportionalto the amount of the frequently updated memory and couldhave latency of tens of ms thus we propose using VM as anew federate container and migration enabler in this paper
3 Dynamic VMFederate Migration Method
In a Cloud computing environment resources are sharedamong multiple users The number and nature of theworkload presented by these users can vary over time [5]In addition large-scale military HLA systems dynamicallychange their computation and communication load duringtheir execution time [1 17] Thus migration of simulationtasks is essential in CSim
For CSim of military HLA there are four basic conceptsincluding entity federate VM and host
4 Mathematical Problems in Engineering
HostFederateEntity VM11n 1n 1n1 1
Figure 1 The mapping relations of entity federate VM and host
Federate 1
Federate 2
Federate 3
Federate N
F 1
F 2
F NEntities
From entities to federateto VM and to host VM migration strategy
VM
VM
VM
VMM
VM VM
HostHost
Host
Host
Host
Host
VM
VM
MMA
Migration command
Figure 2 The strategy of clustering entities to federate to VM and to host
(i) An entity often refers to a simulation object such as atank an airplane and a helicopter or a car
(ii) A federate is the encapsulated HLA form of one ormore entities
(iii) VM is the container of one or more federates becauseVM is the basic service unit of userrsquos request in CSim
(iv) Host is the container of VMs which means one ormore VM could be consolidated to one host mostlydepending on the hostrsquos computation ability
As previously mentioned federate migration is enabledby VM in this paper The mapping relations among the fourconcepts are illustrated in Figure 1
The primary clustering strategy of multiscale HLA simu-lation implementation is shown in Figure 2 where VM basedmigration algorithm and implementation approach will beillustrated
In the following sections we study dynamic VM migra-tion mechanisms Section 31 proposes federatesrsquo distributionand communication architecture Section 32 develops thealgorithm solving the problem when to migrate VMs whatVMs are to be migrated and where the migrating VMs are tobe contained
31 VM Distribution and Communication Architecture InHLA simulations entities are often encapsulated in federateswhich are then located into VMs deployed in differenthosts Each federate executes the simulation by messagecommunication The federate communication architecture isshown in Figure 3
In this architecture each VM encapsulates a local-RTIcomponent (LRC) and there is one server-RTI running on ahost The RTI communication here is hybrid which meansthat HLA global management services including time syn-chronizations are executed by the communications amongserver-RTI and LRC Meanwhile federate to federate com-munications including object instance attribute reflectionsand interaction instance sendingreceiving are executed viaLRC in peer-to-peer mode [44]
The VMFederate distribution architecture is designed asFigure 4 each host is equipped with a VM monitor (VMM)in charge of monitoring the VMsrsquo load and hostsrsquo load In ourCSim every VMM is both a load monitor and a migrationexecutor VMM keeps track of running state of local hostand VMs Also VMM captures the snapshots of local VMsperiodically and then keeps them in a database located inlocal host
Meanwhile one host has a migration management agent(MMA) in charge of monitoring all VMsrsquo periodic statustriggering VM migration procedure when a host is over-loaded selecting the migrated federate sets and target VMsand sending the commandMMAmonitors all VMsrsquo periodicstatus by collecting status data from VMM of each host Thisis a pull manner because VMM pulls data from distributedhosts to MMA The frequency of this monitoring is set as 1sin this paper
32 VM Migration Algorithm Based on the architectureshown in Figures 3 and 4 this section delivers a solutionwhich handles dynamic load imbalance in HLA federationsconsidering both computational and communication costs
Mathematical Problems in Engineering 5
VM VM
VM
Global management Global management
Federate to federatecommunication
Global management
Federate to federate
communication
Federate tofederate
communication
Server-RTI
LRC local RTI component
LRC LRC
LRC
Figure 3 VMFederate communication architecture
VM
Host Host
VM
Host
VM
VM
VMM VMM VMM
MigrationcommandMigration
command
VM snapshots
State monitoring
Periodic sampling
MMA migration mgmt agentLRC local RTI component
Server-RTI
MMALRC
LRC
LRC
LRC
VM and host states
middot middot middot
Figure 4 VMFederate distribution architecture
Let us start from analyzing the utilization and workload ofhosts
321 Hostsrsquo Utilization Threshold 119880119901119895(119905) is the utilization ofhost 119901119895 at time 119905
where 119880cpu(119905) is CPU utilization of host 119901119895 (in percent) and119880mem(119905) is memory utilization of host 119901119895 (in percent) 120572 isa coefficient representing the relative importance betweenCPU utilization and memory utilization As both CPU andmemory are equally important for running VM 120572 is set to05
The utilization threshold host 119901119895 is Th119901119895 or 119896 which isa parameter that allows the adjustment of the effect of themethod the lower 119896 is the higher the possible overload is andthe higher possibility of migration is For the determinationof utilization threshold the following equation is used 119896 =lceil(1 + 120573) sdot ((sum
119899
119895=1119880119901119895)119899)rceil where 119896 is the CPU utilization
threshold for all hosts 119880119901119895 is load of node 119895 119899 is the numberof hosts in the simulation system and 0 le 120573 le 02 is anormalized constant To generate moderate migration herewe set 120573 = 01
Obviously there are two load states for the hosts
(i) If 119880119901119895(119905) ge 119896 host 119901119895 is overloaded
(ii) If 119880119901119895(119905) lt 119896 host 119901119895 is not overloaded
6 Mathematical Problems in Engineering
322 Load of Host and VM At time 119905 load 119871119901119895(119905) of host 119901119895is computed by the following equation
where 119862119895 is the computation capacity of host 119901119895 whichis the frequency of the hostrsquos CPU mapped onto millionsinstructions per second (MIPS) ratings of each core [45]
In this paper we assume that each federate has variableworkload throughout a simulation but we can use prelim-inary experiments to test the maximum loaded VMs of ahost according to their configurations Suppose that all VMsare homogeneously configured and have the same amount ofentities the load 119871V119894119895(119905) (MIPS) of VM V119894119895 at time 119905 is definedas follows
where 119899119895 is the number of VMs in host 119901119895 and the assumptionhere also enables that we canmigrate a set of VMs at one time(see algorithm studied later)
323 Communication Cost In CSim HLA federation fed-erates communicate with each other through the interac-tion class instance and object class instance As discussedin Section 22 we assume the interaction class instance isstochastically sent and object class instance is periodicallyupdated every time step for the correctness of simulationsThe communication bandwidth request (bits) between fed-erate 1198861 and 1198862 is as follows
Comm1198861 1198862 =1
sim steplowast obj ins bytes lowast 8 (4)
where obj ins bytes is the amount of object class instancesbytes exchanged every time step This means the communi-cation cost shown in Figure 5 is the object class instancesrsquorequirements of network bandwidth
Thenwe try to compute the communication cost betweenhost and VM and host and host Figure 5 illustrates theinteractions
In most cases a prerequisite is that VMs in a local hostcan communicate much faster than VMs among differenthostsTherefore wemust consider the two cases separately InFigure 5(a) the solid lines are communications among VMswithin hosts and the dashed lines are the communicationsamong VMs of different hosts After communication merg-ing we can get Figure 5(b) where the c1s represents the sumof communication costs between vm1 and host 119901119895
Therefore the communication cost at time 119905 between theVM 119886119894119896 and the host 119901119895 is defined as follows
With respect to two objectives of dynamic load balancingreducing the load of the overloaded hosts and decreasingthe interhost communication cost a dynamic load balancingmodel is proposed as follows
min 119911 (119905) =119899
sum
119895=1
Comm119901119896119901119895 (119905) st 119880119901119895 (119905) lt Th119901119895 (7)
The objective function 119911(119905) is to minimize the interhostcommunications between host 119901119896 with VM to be migratedand other hosts and the constraint is that each hostrsquoscomputation load is below its threshold
324 Migration Algorithm The migration model above is aNP-hard problem Many researchers have used heuristics tofind the optimal solutions and our approach is influencedby them including the works in [17 26 35 36 39ndash4345ndash48] However compared with existing researches ouralgorithm not only considers the periodical HLA object classcommunication cost but also migrates a set of VMs everytime decreasing the migration procedure latency comparedto most one-by-one federate migration methods
Suppose the overloaded host is 119901119896 and the destinationhost selected by migration management agent (MMA) is 119901119895which has least utilization in the simulation The heuristic isto select a set of VMs from 119901119896 to migrate to 119901119895 in order toreduce the load of 119901119896 and minimize the communication costafter migration The algorithm is illustrated in Algorithm 1
For the proposed algorithm in Algorithm 1 the timecomplexity of Steps 1ndash4 is119874(1198732ave) (119873ave is the average numberof VMs per host) and the time complexity of Step 5 is 119874(119898)Thus the time complexity is 119874(119898 lowast 1198732ave) Moreover thealgorithm is executed with the same frequency of MMAmonitoring all VMsrsquo periodic status that is 1s
Mathematical Problems in Engineering 7
Input VM list withm VMs host list with n hostsOutput Deployment that VM to host (VM119894 host119895 | 119894 isin (1 119898) 119895 isin (1 119899))Algorithm
(1) At time 119905 MMA finds that host 119901119896 is overloaded and needs VMmigration where 119880119901119896 (119905) gt 119896119871mig(119905) = (119880119901119896 cpu(119905) minus 119896) lowast 119862119896 Also 119901119895 is the least loaded host in host list If 119880119901119895 (119905) ge 119896 all hostsare overloaded and this algorithm does not perform migration else if 119880119901119895 (119905) lt 119896MMAsends command to 119901119896 that 119901119895 is its migration destination host(2) Then min119871migrate(119905) (119896 minus 119880119901119895 (119905)) lowast 119862119895 is the largest accepted migration load The largestaccepted VM number is calculated according to 119899mig
119895(119905) = floor(min119871mig(119905) (119896 minus 119880119901119895(119905)) lowast 119862119895(119871V119894119896 (119905)))
(3) Calculate the communication cost CommV119894119896 119901119895 (119905) between every VM of 119901119896 and host 119901119895 and thesum of communication cost CommV119894119896 119901119896minusV119894119896 (119905) between the VM in 119901119896 and the rest VMs in 119901119896 TheVM which has min119894(CommV119894119896 119901119896minusV119894119896 (119905) minus CommV119894119896 119901119895 (119905)) is selected into the VM set 119904119896119895(119905) Then theselected VM is removed from 119901119896 while 119901119895 adds the selected VM Accordingly the communicationrelations of VMsrsquo communication are updated(4) If the number of VMs in 119904119896119895(119905) is less than 119899
mig119895(119905) back to Step 3 Otherwise output its planned
migration set 119904119896119895(119905) of 119901119896(5) If 119871 119904119896119895(119905) le 119871mig(119905) VMM of host 119901119896 and 119901119895 starts the migration
Algorithm 1 Communication cost based VM dynamic migration algorithm
4 Experiment Results and Analysis
41 Experiment Design To validate the effectiveness of theproposed VM based HLA simulation load balancing methodin CSim experiments have been designed and implementedThe simulations were run in a system comprising 2 nodesof Lenovo 8200t 2 nodes of HP 6300 Pro MT 6 nodes ofHP Compaq 8000 Elite CMT and a 100Mbitsec Ethernetconnection among all the nodes The node of Lenovo 8200thad an Intel i7-870 (8 cores) 293GHz CPU and 8G MEMThe node of HP 6300 had an Intel i5-3470 (4 cores) 32 GHzCPU and 4G MEM The node of Compaq 8000 had an IntelCore 2 E8400 (2 cores) 300GHz CPU and 2G MEM
The nodes run a paravirtualized Linux CentOS 56 kernelas a privileged virtual machine on top of the Xen hypervisor401 [30] The guest virtual machines are configured tosingle core and run the same version of the Linux kernel asthat of the privileged one HLA platform was AST-RTI [4950] version 20 performing communication through TCPIPconnections
Moreover as our benchmark a practical HLA armoredforce game for tactical training was developed The gamecoded in CC++ was used to conduct experiments and ana-lyze the performance of our approach The scenario for ourexperiments was a simulation of battle engagement game ofred and blue tank forces which were hierarchically organizedas Platoon (P) Company (C) Battalion (B) and Regiment(R) The tank effectuated random selection of several tacticalroutes and engagement strategies in two-dimensional spacethat was within range of some military training location
The organization structure of tank forces is illustratedin Figure 6 which shows that red forces are formed hierar-chically in 3 to 3 organization This means that every redcompany has 3 platoons and every platoon has 3 tanks whilefor the blue side it is formed in 4 to 4 organization which
Table 1 The number of VMs in different game scenarios
Scenario (or scale) Numberof VMs
1 red Company versus 1 blue Company 91 red Battalion versus 1 blue Company 181 red Company versus 1 blue Battalion 251 red Battalion versus 1 blue Battalion 341 red Battalion + 1 red Company versus 1 blue Battalion 381 red Battalion + 1 red Company versus 1 blue Battalion+ 1 blue Company 43
1 red Battalion + 2 red Company versus 1 blue Battalion+ 1 blue company 47
1 red Battalion + 2 red Company versus 1 blue Battalion+ 2 blue Company 52
2 red Battalion versus 1 blue Battalion + 2 blueCompany 57
means every blue company has 4 platoons and every platoonhas 4 tanks
In order to accomplish such simulations we cluster tankentities into VMs according to their military affiliations Theabbreviations are P Platoon C Company B Battalion RRegiment r red b blue
Table 1 shows the experimentsrsquo deployment Each VMcontains one federate in the experiments because computa-tion and communication costs are mainly due to the numberof tank entities When the number of entities in one VM isfixed the number of federates has little impact on the VMrsquoscosts as interhost communication cost is normally muchgreater than local host cost
Moreover each red tank Company is deployed with4VMs which are Platoon-1 (P-1) P-2 P-3 and Company
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
Meanwhile one of the most important MampS standardshigh level architecture (HLA) is an essential IEEE simu-lation interoperability standard in the area of distributedand parallel simulations And IEEE has developed HLAimplementation standards such as HLA-evolved [7ndash9] WithHLA and its implementation program runtime infrastructure(RTI) a lot of HLA compatible simulation systems have beendeveloped including large-scale military HLA applicationssuch as ModSAF [10] CCTT SAF [11] OneSAF [12 13]and COSIM [14] These large-scale simulations are usuallyhardware-consuming and software-intensive as they containnumerous models of environments and entities Grid tech-nology [15ndash17] has been used to solve the resource allocationand scheduling problem but Grid is not fully used whilecloud is more supported by many international companiessuch as Amazon [18] Google [19] and Softlayer [20]
As such we propose to use Cloud computing techniquesto develop simulation systems to solve the high cost andoverlapped hardware and software resources problem withcloud Cloud computing infrastructure offers numerous ben-efits such as reconfigurable dynamic resources and unifiedand simplified access to resources
Although the works in [1ndash6] explored new paradigmsof CSim few papers addressed the specific steps of how torunHLA simulations in cloud especially virtualization basedlarge-scale military applications Therefore in this paperwe mainly study how to run existing large-scale HLARTIsimulations with cloud and how to run themmore efficiently
The paper is structured as follows In Section 2 keytechnology of Cloud computing virtualization and HLAload balancing strategies are introduced and analyzed Corepart of the contribution Section 3 introduces the proposedvirtualization based HLA federation framework and run-time dynamicmigration algorithm Section 4 shows designedvalidation experiments and discusses results Finally conclu-sions are drawn and future works are discussed
2 Related Work
In this section virtualization related technologies and loadbalancing approaches are studied and analyzed
21 Virtualization Technology and HLA Simulations Incloud Virtualization technology (VT) encapsulates the sim-ulation application models and software packages to a rela-tively independent execution environment VM and deploysVMs to the underlying Physical Machines (PM or hosts) [21]Furthermore by the segmentation in space and the time-sharing in time [22 23] VT encapsulates the distribution andheterogeneity of the hardware resources allowing multiplevirtual machines to run simultaneously on one host as wellas the migration of VMs among different hosts In this wayVT greatly improves the resource utilization and quality ofservice and is gradually implemented in the medium-scaleand large-scale distributed simulation platform
One of the key virtualization technologies is VMmonitor(VMM) which connects applications and VMs with under-lying hardware to control the running of multiple operatingsystems VMM decouples the software from the hardware by
inserting a layer of interaction between the software runningin the VM and hardware so that VMM could obtain directcontrol of how operating systems in VMs access underlyinghardware resources [24] The design of VMM also makesit possible to implement the functions like live migration[25] and security that are difficult to achieve in modernoperating systemsThrough VMM computing resources canbe encapsulated into a collection of VMs according to userrsquosrequirements and VMs can be remapped and replicatedeasily In addition the state of VMs can be suspendedand resumed at any time and the running context can berolled back to a checkpoint Thus virtualization technologyprovides the ability to consolidatemanagement of distributedand heterogeneous resources by means of decoupling soft-ware and hardware transparent access encapsulation livemigration and resources isolation
Compared with the federate migration method of [1726] the main difference is that we use Virtual Machines(VMs) to be containers of federates From the perspective ofpractical implementation using cloud-based VMs to handlethe resource management and dynamic migration of HLAsimulation systems has the following merits compared withprevious approaches
(i) It saves programming efforts to implement migrationof federates with the support of the virtualizationmiddleware interfaces developed by many businessor open-source VM vendors such as VMware [27]Opennebula [28] Eucalyptus [29] and Xen [30]
(ii) VM live (or online) migration technique makes itfeasible to realize high efficient federate migrationPrevious works show that VM live migration down-time is mainly proportional to the amount of theldquowritable working setrdquo and could have downtime aslow as 60 to 200ms [31 32]
(iii) VM encapsulating HLA application software pack-ages such as RTI could be copied easily This greatlysaves the time and efforts simulation practitionersspend to configure the operational system and soft-ware environment which is often a tricky and tediousprocess especially during the large-scale simulationapplications deployment phase
Therefore it is more convenient for simulationists todevelop and run large-scale simulations if we adapt thevirtualization technology That is why we study cloud-basedHLA simulation in this paper However when we are ensuredthat we could use VMs to help us handle the federatemigrations we also encounter the problems that using VM toencapsulate federates create additional overhead Someworks[1 33 34] have designed experiments to find performanceloss caused by virtualizationThey found that the applications(with different event interaction complexity) ran in VM havethe maximum loss of 333 compared to ran in PhysicalMachine Although the performance loss is not high westill have to take more measures to make HLA federationrun faster One effective way is to improve current loadbalancing algorithms with better heuristics which considersHLA communication cost among federates
Mathematical Problems in Engineering 3
22 Load Balance Strategy in HLA and Beyond A famousdeveloping IEEE standard in Modeling and Simulation(MampS) society HLA is also gradually known for its slowrunning characteristics One important reason is that HLAis short of load-balance methods [17 26 35ndash38] to make theheavy load host run faster while all the other nodes must waituntil the slowest federate advances so they can move to thenext safe lookahead
Current studies of HLA load balance techniques could bedivided into two categories
(i) Load Balance Algorithms for Distributed and Parallel Sys-tems Including HLA Load balance strategies for distributedand parallel systems are basically adaptable for HLA whichis designed to provide a common set of framework andrules for distributed simulation In load balance algorithmsthe core concerns are computing and communication costs[17 26 39ndash43] Our algorithm has been influenced by earlierworks in three main areas heuristics computation andcommunication cost
Most dynamic load balancing works use heuristic algo-rithms because of its NP-hard nature [39] Paper [40] studiedheuristic methods for dynamic load balancing in a message-passing supercomputer It uses easy-to-implement heuristics(choosing a node with minimum load as the migration desti-nation node) and variable threshold in migrating processesamong the multicomputer nodes Genetic algorithms areused in [41] to dynamically distribute simulation tasks andcompared with a first fit algorithm and a random allocationscheme In this work the test results show that geneticmethod outperforms the other two both in completion timeand average processor utilization In paper [39] adaptiveload threshold is emphasized to suit the changing load onthe system and its basic strategy is also to ensure thatheavily loaded node is balanced first with lightly loaded nodeSimulated annealing algorithm is addressed in paper [42]which transforms the problem into a NP-complete problemnamed degree-constrained minimum spanning tree Thisapproach needs to adjust the heuristic factors according tothe practical applications These works use heuristics as anessential approach which is also our basic strategy becauseit is simple practical and time-saving especially in large-scale distributed simulation However most of the traditionalworks are one-by-one migration while in our approach it isdesigned to migrate a set of interaction-aware VMs to savethe cost of migration procedure latency and communication
Moreover papers [17 26] study dynamic load balancingusing Grid services for HLA-based large-scale simulationsThese pioneering works store all the resources in a queue andorganize them in ascendant order and themigration pairs areassembled with both extremities The algorithm is practicaland simple but it does not consider the communication costFujimoto et al in [4 5] addressed ldquoNetwork traffic and com-munication delays are significant in current implementationof Cloud computing infrastructurerdquo Thus for the cloud-based large-scale simulations communication cost must betaken into account in dynamic HLA load balancing
All the abovementioned works are meaningful but HLAsimulation systems have some more distinct characteristics
which are neglected by these researches For instance allinteractions among HLA federates are mainly modeled andimplemented with interaction classes and object classes Inmost cases of military simulations the former classes arestochastic and the latter classes are periodic For example thestatus data describing the position of a tank in Cartesiancoordinates would be modeled as an object class and itsinstance data is updated by the tank entity at every timestep The data describing the fire of a tank entity would bemodeled as an interaction class and its instance data is sentwhen the tank fires Moreover the interaction classes objectclasses and their relations of subscription and publicationare all defined before the simulation starts Therefore despitethe stochastic interaction class communications we can stillcompute the communication cost of periodic object classesand these core HLA characteristics will be considered in theload balance strategy of this paper
(ii) Federate Migration Techniques The approach presentedby Tan et al [35 36] accomplishes federate migration witha federate wrapper which controls the federates executionthrough SugarCubes stopping and resuming a federate Thekey point of this research is using the third-partymechanismsto avoid HLA federation saverestore approach Althoughit gains less latency than some other works its minimumlatency is about 8 seconds which is much longer than VMapproach in this paper
To propose and implement an efficient procedure forbalancing HLA simulationrsquos load paper [17] migrates feder-ates through the GRAM Gridsrsquo service such as Web servicesgrid resource allocation and management (WS GRAM)and GridFTP To make the migration delay as negligibleas possible the authors implement the migration in twosteps In the first step it executes in a way that federatedoes not stop its execution while initialization files aretransferred In the second step the RTI methods are calledto freeze the federation and the rest of data regarding thefederatersquos running status and messages are transferred Thetwo migration steps are well-devised and similar strategy isimplemented in this paper while using live VMmigration
Furthermore in the companion work of [17] thesis [26]decreases the migration latency to around 08 seconds whichis highly efficiency However as we described in the previoussection VM live migration downtime is mainly proportionalto the amount of the frequently updated memory and couldhave latency of tens of ms thus we propose using VM as anew federate container and migration enabler in this paper
3 Dynamic VMFederate Migration Method
In a Cloud computing environment resources are sharedamong multiple users The number and nature of theworkload presented by these users can vary over time [5]In addition large-scale military HLA systems dynamicallychange their computation and communication load duringtheir execution time [1 17] Thus migration of simulationtasks is essential in CSim
For CSim of military HLA there are four basic conceptsincluding entity federate VM and host
4 Mathematical Problems in Engineering
HostFederateEntity VM11n 1n 1n1 1
Figure 1 The mapping relations of entity federate VM and host
Federate 1
Federate 2
Federate 3
Federate N
F 1
F 2
F NEntities
From entities to federateto VM and to host VM migration strategy
VM
VM
VM
VMM
VM VM
HostHost
Host
Host
Host
Host
VM
VM
MMA
Migration command
Figure 2 The strategy of clustering entities to federate to VM and to host
(i) An entity often refers to a simulation object such as atank an airplane and a helicopter or a car
(ii) A federate is the encapsulated HLA form of one ormore entities
(iii) VM is the container of one or more federates becauseVM is the basic service unit of userrsquos request in CSim
(iv) Host is the container of VMs which means one ormore VM could be consolidated to one host mostlydepending on the hostrsquos computation ability
As previously mentioned federate migration is enabledby VM in this paper The mapping relations among the fourconcepts are illustrated in Figure 1
The primary clustering strategy of multiscale HLA simu-lation implementation is shown in Figure 2 where VM basedmigration algorithm and implementation approach will beillustrated
In the following sections we study dynamic VM migra-tion mechanisms Section 31 proposes federatesrsquo distributionand communication architecture Section 32 develops thealgorithm solving the problem when to migrate VMs whatVMs are to be migrated and where the migrating VMs are tobe contained
31 VM Distribution and Communication Architecture InHLA simulations entities are often encapsulated in federateswhich are then located into VMs deployed in differenthosts Each federate executes the simulation by messagecommunication The federate communication architecture isshown in Figure 3
In this architecture each VM encapsulates a local-RTIcomponent (LRC) and there is one server-RTI running on ahost The RTI communication here is hybrid which meansthat HLA global management services including time syn-chronizations are executed by the communications amongserver-RTI and LRC Meanwhile federate to federate com-munications including object instance attribute reflectionsand interaction instance sendingreceiving are executed viaLRC in peer-to-peer mode [44]
The VMFederate distribution architecture is designed asFigure 4 each host is equipped with a VM monitor (VMM)in charge of monitoring the VMsrsquo load and hostsrsquo load In ourCSim every VMM is both a load monitor and a migrationexecutor VMM keeps track of running state of local hostand VMs Also VMM captures the snapshots of local VMsperiodically and then keeps them in a database located inlocal host
Meanwhile one host has a migration management agent(MMA) in charge of monitoring all VMsrsquo periodic statustriggering VM migration procedure when a host is over-loaded selecting the migrated federate sets and target VMsand sending the commandMMAmonitors all VMsrsquo periodicstatus by collecting status data from VMM of each host Thisis a pull manner because VMM pulls data from distributedhosts to MMA The frequency of this monitoring is set as 1sin this paper
32 VM Migration Algorithm Based on the architectureshown in Figures 3 and 4 this section delivers a solutionwhich handles dynamic load imbalance in HLA federationsconsidering both computational and communication costs
Mathematical Problems in Engineering 5
VM VM
VM
Global management Global management
Federate to federatecommunication
Global management
Federate to federate
communication
Federate tofederate
communication
Server-RTI
LRC local RTI component
LRC LRC
LRC
Figure 3 VMFederate communication architecture
VM
Host Host
VM
Host
VM
VM
VMM VMM VMM
MigrationcommandMigration
command
VM snapshots
State monitoring
Periodic sampling
MMA migration mgmt agentLRC local RTI component
Server-RTI
MMALRC
LRC
LRC
LRC
VM and host states
middot middot middot
Figure 4 VMFederate distribution architecture
Let us start from analyzing the utilization and workload ofhosts
321 Hostsrsquo Utilization Threshold 119880119901119895(119905) is the utilization ofhost 119901119895 at time 119905
where 119880cpu(119905) is CPU utilization of host 119901119895 (in percent) and119880mem(119905) is memory utilization of host 119901119895 (in percent) 120572 isa coefficient representing the relative importance betweenCPU utilization and memory utilization As both CPU andmemory are equally important for running VM 120572 is set to05
The utilization threshold host 119901119895 is Th119901119895 or 119896 which isa parameter that allows the adjustment of the effect of themethod the lower 119896 is the higher the possible overload is andthe higher possibility of migration is For the determinationof utilization threshold the following equation is used 119896 =lceil(1 + 120573) sdot ((sum
119899
119895=1119880119901119895)119899)rceil where 119896 is the CPU utilization
threshold for all hosts 119880119901119895 is load of node 119895 119899 is the numberof hosts in the simulation system and 0 le 120573 le 02 is anormalized constant To generate moderate migration herewe set 120573 = 01
Obviously there are two load states for the hosts
(i) If 119880119901119895(119905) ge 119896 host 119901119895 is overloaded
(ii) If 119880119901119895(119905) lt 119896 host 119901119895 is not overloaded
6 Mathematical Problems in Engineering
322 Load of Host and VM At time 119905 load 119871119901119895(119905) of host 119901119895is computed by the following equation
where 119862119895 is the computation capacity of host 119901119895 whichis the frequency of the hostrsquos CPU mapped onto millionsinstructions per second (MIPS) ratings of each core [45]
In this paper we assume that each federate has variableworkload throughout a simulation but we can use prelim-inary experiments to test the maximum loaded VMs of ahost according to their configurations Suppose that all VMsare homogeneously configured and have the same amount ofentities the load 119871V119894119895(119905) (MIPS) of VM V119894119895 at time 119905 is definedas follows
where 119899119895 is the number of VMs in host 119901119895 and the assumptionhere also enables that we canmigrate a set of VMs at one time(see algorithm studied later)
323 Communication Cost In CSim HLA federation fed-erates communicate with each other through the interac-tion class instance and object class instance As discussedin Section 22 we assume the interaction class instance isstochastically sent and object class instance is periodicallyupdated every time step for the correctness of simulationsThe communication bandwidth request (bits) between fed-erate 1198861 and 1198862 is as follows
Comm1198861 1198862 =1
sim steplowast obj ins bytes lowast 8 (4)
where obj ins bytes is the amount of object class instancesbytes exchanged every time step This means the communi-cation cost shown in Figure 5 is the object class instancesrsquorequirements of network bandwidth
Thenwe try to compute the communication cost betweenhost and VM and host and host Figure 5 illustrates theinteractions
In most cases a prerequisite is that VMs in a local hostcan communicate much faster than VMs among differenthostsTherefore wemust consider the two cases separately InFigure 5(a) the solid lines are communications among VMswithin hosts and the dashed lines are the communicationsamong VMs of different hosts After communication merg-ing we can get Figure 5(b) where the c1s represents the sumof communication costs between vm1 and host 119901119895
Therefore the communication cost at time 119905 between theVM 119886119894119896 and the host 119901119895 is defined as follows
With respect to two objectives of dynamic load balancingreducing the load of the overloaded hosts and decreasingthe interhost communication cost a dynamic load balancingmodel is proposed as follows
min 119911 (119905) =119899
sum
119895=1
Comm119901119896119901119895 (119905) st 119880119901119895 (119905) lt Th119901119895 (7)
The objective function 119911(119905) is to minimize the interhostcommunications between host 119901119896 with VM to be migratedand other hosts and the constraint is that each hostrsquoscomputation load is below its threshold
324 Migration Algorithm The migration model above is aNP-hard problem Many researchers have used heuristics tofind the optimal solutions and our approach is influencedby them including the works in [17 26 35 36 39ndash4345ndash48] However compared with existing researches ouralgorithm not only considers the periodical HLA object classcommunication cost but also migrates a set of VMs everytime decreasing the migration procedure latency comparedto most one-by-one federate migration methods
Suppose the overloaded host is 119901119896 and the destinationhost selected by migration management agent (MMA) is 119901119895which has least utilization in the simulation The heuristic isto select a set of VMs from 119901119896 to migrate to 119901119895 in order toreduce the load of 119901119896 and minimize the communication costafter migration The algorithm is illustrated in Algorithm 1
For the proposed algorithm in Algorithm 1 the timecomplexity of Steps 1ndash4 is119874(1198732ave) (119873ave is the average numberof VMs per host) and the time complexity of Step 5 is 119874(119898)Thus the time complexity is 119874(119898 lowast 1198732ave) Moreover thealgorithm is executed with the same frequency of MMAmonitoring all VMsrsquo periodic status that is 1s
Mathematical Problems in Engineering 7
Input VM list withm VMs host list with n hostsOutput Deployment that VM to host (VM119894 host119895 | 119894 isin (1 119898) 119895 isin (1 119899))Algorithm
(1) At time 119905 MMA finds that host 119901119896 is overloaded and needs VMmigration where 119880119901119896 (119905) gt 119896119871mig(119905) = (119880119901119896 cpu(119905) minus 119896) lowast 119862119896 Also 119901119895 is the least loaded host in host list If 119880119901119895 (119905) ge 119896 all hostsare overloaded and this algorithm does not perform migration else if 119880119901119895 (119905) lt 119896MMAsends command to 119901119896 that 119901119895 is its migration destination host(2) Then min119871migrate(119905) (119896 minus 119880119901119895 (119905)) lowast 119862119895 is the largest accepted migration load The largestaccepted VM number is calculated according to 119899mig
119895(119905) = floor(min119871mig(119905) (119896 minus 119880119901119895(119905)) lowast 119862119895(119871V119894119896 (119905)))
(3) Calculate the communication cost CommV119894119896 119901119895 (119905) between every VM of 119901119896 and host 119901119895 and thesum of communication cost CommV119894119896 119901119896minusV119894119896 (119905) between the VM in 119901119896 and the rest VMs in 119901119896 TheVM which has min119894(CommV119894119896 119901119896minusV119894119896 (119905) minus CommV119894119896 119901119895 (119905)) is selected into the VM set 119904119896119895(119905) Then theselected VM is removed from 119901119896 while 119901119895 adds the selected VM Accordingly the communicationrelations of VMsrsquo communication are updated(4) If the number of VMs in 119904119896119895(119905) is less than 119899
mig119895(119905) back to Step 3 Otherwise output its planned
migration set 119904119896119895(119905) of 119901119896(5) If 119871 119904119896119895(119905) le 119871mig(119905) VMM of host 119901119896 and 119901119895 starts the migration
Algorithm 1 Communication cost based VM dynamic migration algorithm
4 Experiment Results and Analysis
41 Experiment Design To validate the effectiveness of theproposed VM based HLA simulation load balancing methodin CSim experiments have been designed and implementedThe simulations were run in a system comprising 2 nodesof Lenovo 8200t 2 nodes of HP 6300 Pro MT 6 nodes ofHP Compaq 8000 Elite CMT and a 100Mbitsec Ethernetconnection among all the nodes The node of Lenovo 8200thad an Intel i7-870 (8 cores) 293GHz CPU and 8G MEMThe node of HP 6300 had an Intel i5-3470 (4 cores) 32 GHzCPU and 4G MEM The node of Compaq 8000 had an IntelCore 2 E8400 (2 cores) 300GHz CPU and 2G MEM
The nodes run a paravirtualized Linux CentOS 56 kernelas a privileged virtual machine on top of the Xen hypervisor401 [30] The guest virtual machines are configured tosingle core and run the same version of the Linux kernel asthat of the privileged one HLA platform was AST-RTI [4950] version 20 performing communication through TCPIPconnections
Moreover as our benchmark a practical HLA armoredforce game for tactical training was developed The gamecoded in CC++ was used to conduct experiments and ana-lyze the performance of our approach The scenario for ourexperiments was a simulation of battle engagement game ofred and blue tank forces which were hierarchically organizedas Platoon (P) Company (C) Battalion (B) and Regiment(R) The tank effectuated random selection of several tacticalroutes and engagement strategies in two-dimensional spacethat was within range of some military training location
The organization structure of tank forces is illustratedin Figure 6 which shows that red forces are formed hierar-chically in 3 to 3 organization This means that every redcompany has 3 platoons and every platoon has 3 tanks whilefor the blue side it is formed in 4 to 4 organization which
Table 1 The number of VMs in different game scenarios
Scenario (or scale) Numberof VMs
1 red Company versus 1 blue Company 91 red Battalion versus 1 blue Company 181 red Company versus 1 blue Battalion 251 red Battalion versus 1 blue Battalion 341 red Battalion + 1 red Company versus 1 blue Battalion 381 red Battalion + 1 red Company versus 1 blue Battalion+ 1 blue Company 43
1 red Battalion + 2 red Company versus 1 blue Battalion+ 1 blue company 47
1 red Battalion + 2 red Company versus 1 blue Battalion+ 2 blue Company 52
2 red Battalion versus 1 blue Battalion + 2 blueCompany 57
means every blue company has 4 platoons and every platoonhas 4 tanks
In order to accomplish such simulations we cluster tankentities into VMs according to their military affiliations Theabbreviations are P Platoon C Company B Battalion RRegiment r red b blue
Table 1 shows the experimentsrsquo deployment Each VMcontains one federate in the experiments because computa-tion and communication costs are mainly due to the numberof tank entities When the number of entities in one VM isfixed the number of federates has little impact on the VMrsquoscosts as interhost communication cost is normally muchgreater than local host cost
Moreover each red tank Company is deployed with4VMs which are Platoon-1 (P-1) P-2 P-3 and Company
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
22 Load Balance Strategy in HLA and Beyond A famousdeveloping IEEE standard in Modeling and Simulation(MampS) society HLA is also gradually known for its slowrunning characteristics One important reason is that HLAis short of load-balance methods [17 26 35ndash38] to make theheavy load host run faster while all the other nodes must waituntil the slowest federate advances so they can move to thenext safe lookahead
Current studies of HLA load balance techniques could bedivided into two categories
(i) Load Balance Algorithms for Distributed and Parallel Sys-tems Including HLA Load balance strategies for distributedand parallel systems are basically adaptable for HLA whichis designed to provide a common set of framework andrules for distributed simulation In load balance algorithmsthe core concerns are computing and communication costs[17 26 39ndash43] Our algorithm has been influenced by earlierworks in three main areas heuristics computation andcommunication cost
Most dynamic load balancing works use heuristic algo-rithms because of its NP-hard nature [39] Paper [40] studiedheuristic methods for dynamic load balancing in a message-passing supercomputer It uses easy-to-implement heuristics(choosing a node with minimum load as the migration desti-nation node) and variable threshold in migrating processesamong the multicomputer nodes Genetic algorithms areused in [41] to dynamically distribute simulation tasks andcompared with a first fit algorithm and a random allocationscheme In this work the test results show that geneticmethod outperforms the other two both in completion timeand average processor utilization In paper [39] adaptiveload threshold is emphasized to suit the changing load onthe system and its basic strategy is also to ensure thatheavily loaded node is balanced first with lightly loaded nodeSimulated annealing algorithm is addressed in paper [42]which transforms the problem into a NP-complete problemnamed degree-constrained minimum spanning tree Thisapproach needs to adjust the heuristic factors according tothe practical applications These works use heuristics as anessential approach which is also our basic strategy becauseit is simple practical and time-saving especially in large-scale distributed simulation However most of the traditionalworks are one-by-one migration while in our approach it isdesigned to migrate a set of interaction-aware VMs to savethe cost of migration procedure latency and communication
Moreover papers [17 26] study dynamic load balancingusing Grid services for HLA-based large-scale simulationsThese pioneering works store all the resources in a queue andorganize them in ascendant order and themigration pairs areassembled with both extremities The algorithm is practicaland simple but it does not consider the communication costFujimoto et al in [4 5] addressed ldquoNetwork traffic and com-munication delays are significant in current implementationof Cloud computing infrastructurerdquo Thus for the cloud-based large-scale simulations communication cost must betaken into account in dynamic HLA load balancing
All the abovementioned works are meaningful but HLAsimulation systems have some more distinct characteristics
which are neglected by these researches For instance allinteractions among HLA federates are mainly modeled andimplemented with interaction classes and object classes Inmost cases of military simulations the former classes arestochastic and the latter classes are periodic For example thestatus data describing the position of a tank in Cartesiancoordinates would be modeled as an object class and itsinstance data is updated by the tank entity at every timestep The data describing the fire of a tank entity would bemodeled as an interaction class and its instance data is sentwhen the tank fires Moreover the interaction classes objectclasses and their relations of subscription and publicationare all defined before the simulation starts Therefore despitethe stochastic interaction class communications we can stillcompute the communication cost of periodic object classesand these core HLA characteristics will be considered in theload balance strategy of this paper
(ii) Federate Migration Techniques The approach presentedby Tan et al [35 36] accomplishes federate migration witha federate wrapper which controls the federates executionthrough SugarCubes stopping and resuming a federate Thekey point of this research is using the third-partymechanismsto avoid HLA federation saverestore approach Althoughit gains less latency than some other works its minimumlatency is about 8 seconds which is much longer than VMapproach in this paper
To propose and implement an efficient procedure forbalancing HLA simulationrsquos load paper [17] migrates feder-ates through the GRAM Gridsrsquo service such as Web servicesgrid resource allocation and management (WS GRAM)and GridFTP To make the migration delay as negligibleas possible the authors implement the migration in twosteps In the first step it executes in a way that federatedoes not stop its execution while initialization files aretransferred In the second step the RTI methods are calledto freeze the federation and the rest of data regarding thefederatersquos running status and messages are transferred Thetwo migration steps are well-devised and similar strategy isimplemented in this paper while using live VMmigration
Furthermore in the companion work of [17] thesis [26]decreases the migration latency to around 08 seconds whichis highly efficiency However as we described in the previoussection VM live migration downtime is mainly proportionalto the amount of the frequently updated memory and couldhave latency of tens of ms thus we propose using VM as anew federate container and migration enabler in this paper
3 Dynamic VMFederate Migration Method
In a Cloud computing environment resources are sharedamong multiple users The number and nature of theworkload presented by these users can vary over time [5]In addition large-scale military HLA systems dynamicallychange their computation and communication load duringtheir execution time [1 17] Thus migration of simulationtasks is essential in CSim
For CSim of military HLA there are four basic conceptsincluding entity federate VM and host
4 Mathematical Problems in Engineering
HostFederateEntity VM11n 1n 1n1 1
Figure 1 The mapping relations of entity federate VM and host
Federate 1
Federate 2
Federate 3
Federate N
F 1
F 2
F NEntities
From entities to federateto VM and to host VM migration strategy
VM
VM
VM
VMM
VM VM
HostHost
Host
Host
Host
Host
VM
VM
MMA
Migration command
Figure 2 The strategy of clustering entities to federate to VM and to host
(i) An entity often refers to a simulation object such as atank an airplane and a helicopter or a car
(ii) A federate is the encapsulated HLA form of one ormore entities
(iii) VM is the container of one or more federates becauseVM is the basic service unit of userrsquos request in CSim
(iv) Host is the container of VMs which means one ormore VM could be consolidated to one host mostlydepending on the hostrsquos computation ability
As previously mentioned federate migration is enabledby VM in this paper The mapping relations among the fourconcepts are illustrated in Figure 1
The primary clustering strategy of multiscale HLA simu-lation implementation is shown in Figure 2 where VM basedmigration algorithm and implementation approach will beillustrated
In the following sections we study dynamic VM migra-tion mechanisms Section 31 proposes federatesrsquo distributionand communication architecture Section 32 develops thealgorithm solving the problem when to migrate VMs whatVMs are to be migrated and where the migrating VMs are tobe contained
31 VM Distribution and Communication Architecture InHLA simulations entities are often encapsulated in federateswhich are then located into VMs deployed in differenthosts Each federate executes the simulation by messagecommunication The federate communication architecture isshown in Figure 3
In this architecture each VM encapsulates a local-RTIcomponent (LRC) and there is one server-RTI running on ahost The RTI communication here is hybrid which meansthat HLA global management services including time syn-chronizations are executed by the communications amongserver-RTI and LRC Meanwhile federate to federate com-munications including object instance attribute reflectionsand interaction instance sendingreceiving are executed viaLRC in peer-to-peer mode [44]
The VMFederate distribution architecture is designed asFigure 4 each host is equipped with a VM monitor (VMM)in charge of monitoring the VMsrsquo load and hostsrsquo load In ourCSim every VMM is both a load monitor and a migrationexecutor VMM keeps track of running state of local hostand VMs Also VMM captures the snapshots of local VMsperiodically and then keeps them in a database located inlocal host
Meanwhile one host has a migration management agent(MMA) in charge of monitoring all VMsrsquo periodic statustriggering VM migration procedure when a host is over-loaded selecting the migrated federate sets and target VMsand sending the commandMMAmonitors all VMsrsquo periodicstatus by collecting status data from VMM of each host Thisis a pull manner because VMM pulls data from distributedhosts to MMA The frequency of this monitoring is set as 1sin this paper
32 VM Migration Algorithm Based on the architectureshown in Figures 3 and 4 this section delivers a solutionwhich handles dynamic load imbalance in HLA federationsconsidering both computational and communication costs
Mathematical Problems in Engineering 5
VM VM
VM
Global management Global management
Federate to federatecommunication
Global management
Federate to federate
communication
Federate tofederate
communication
Server-RTI
LRC local RTI component
LRC LRC
LRC
Figure 3 VMFederate communication architecture
VM
Host Host
VM
Host
VM
VM
VMM VMM VMM
MigrationcommandMigration
command
VM snapshots
State monitoring
Periodic sampling
MMA migration mgmt agentLRC local RTI component
Server-RTI
MMALRC
LRC
LRC
LRC
VM and host states
middot middot middot
Figure 4 VMFederate distribution architecture
Let us start from analyzing the utilization and workload ofhosts
321 Hostsrsquo Utilization Threshold 119880119901119895(119905) is the utilization ofhost 119901119895 at time 119905
where 119880cpu(119905) is CPU utilization of host 119901119895 (in percent) and119880mem(119905) is memory utilization of host 119901119895 (in percent) 120572 isa coefficient representing the relative importance betweenCPU utilization and memory utilization As both CPU andmemory are equally important for running VM 120572 is set to05
The utilization threshold host 119901119895 is Th119901119895 or 119896 which isa parameter that allows the adjustment of the effect of themethod the lower 119896 is the higher the possible overload is andthe higher possibility of migration is For the determinationof utilization threshold the following equation is used 119896 =lceil(1 + 120573) sdot ((sum
119899
119895=1119880119901119895)119899)rceil where 119896 is the CPU utilization
threshold for all hosts 119880119901119895 is load of node 119895 119899 is the numberof hosts in the simulation system and 0 le 120573 le 02 is anormalized constant To generate moderate migration herewe set 120573 = 01
Obviously there are two load states for the hosts
(i) If 119880119901119895(119905) ge 119896 host 119901119895 is overloaded
(ii) If 119880119901119895(119905) lt 119896 host 119901119895 is not overloaded
6 Mathematical Problems in Engineering
322 Load of Host and VM At time 119905 load 119871119901119895(119905) of host 119901119895is computed by the following equation
where 119862119895 is the computation capacity of host 119901119895 whichis the frequency of the hostrsquos CPU mapped onto millionsinstructions per second (MIPS) ratings of each core [45]
In this paper we assume that each federate has variableworkload throughout a simulation but we can use prelim-inary experiments to test the maximum loaded VMs of ahost according to their configurations Suppose that all VMsare homogeneously configured and have the same amount ofentities the load 119871V119894119895(119905) (MIPS) of VM V119894119895 at time 119905 is definedas follows
where 119899119895 is the number of VMs in host 119901119895 and the assumptionhere also enables that we canmigrate a set of VMs at one time(see algorithm studied later)
323 Communication Cost In CSim HLA federation fed-erates communicate with each other through the interac-tion class instance and object class instance As discussedin Section 22 we assume the interaction class instance isstochastically sent and object class instance is periodicallyupdated every time step for the correctness of simulationsThe communication bandwidth request (bits) between fed-erate 1198861 and 1198862 is as follows
Comm1198861 1198862 =1
sim steplowast obj ins bytes lowast 8 (4)
where obj ins bytes is the amount of object class instancesbytes exchanged every time step This means the communi-cation cost shown in Figure 5 is the object class instancesrsquorequirements of network bandwidth
Thenwe try to compute the communication cost betweenhost and VM and host and host Figure 5 illustrates theinteractions
In most cases a prerequisite is that VMs in a local hostcan communicate much faster than VMs among differenthostsTherefore wemust consider the two cases separately InFigure 5(a) the solid lines are communications among VMswithin hosts and the dashed lines are the communicationsamong VMs of different hosts After communication merg-ing we can get Figure 5(b) where the c1s represents the sumof communication costs between vm1 and host 119901119895
Therefore the communication cost at time 119905 between theVM 119886119894119896 and the host 119901119895 is defined as follows
With respect to two objectives of dynamic load balancingreducing the load of the overloaded hosts and decreasingthe interhost communication cost a dynamic load balancingmodel is proposed as follows
min 119911 (119905) =119899
sum
119895=1
Comm119901119896119901119895 (119905) st 119880119901119895 (119905) lt Th119901119895 (7)
The objective function 119911(119905) is to minimize the interhostcommunications between host 119901119896 with VM to be migratedand other hosts and the constraint is that each hostrsquoscomputation load is below its threshold
324 Migration Algorithm The migration model above is aNP-hard problem Many researchers have used heuristics tofind the optimal solutions and our approach is influencedby them including the works in [17 26 35 36 39ndash4345ndash48] However compared with existing researches ouralgorithm not only considers the periodical HLA object classcommunication cost but also migrates a set of VMs everytime decreasing the migration procedure latency comparedto most one-by-one federate migration methods
Suppose the overloaded host is 119901119896 and the destinationhost selected by migration management agent (MMA) is 119901119895which has least utilization in the simulation The heuristic isto select a set of VMs from 119901119896 to migrate to 119901119895 in order toreduce the load of 119901119896 and minimize the communication costafter migration The algorithm is illustrated in Algorithm 1
For the proposed algorithm in Algorithm 1 the timecomplexity of Steps 1ndash4 is119874(1198732ave) (119873ave is the average numberof VMs per host) and the time complexity of Step 5 is 119874(119898)Thus the time complexity is 119874(119898 lowast 1198732ave) Moreover thealgorithm is executed with the same frequency of MMAmonitoring all VMsrsquo periodic status that is 1s
Mathematical Problems in Engineering 7
Input VM list withm VMs host list with n hostsOutput Deployment that VM to host (VM119894 host119895 | 119894 isin (1 119898) 119895 isin (1 119899))Algorithm
(1) At time 119905 MMA finds that host 119901119896 is overloaded and needs VMmigration where 119880119901119896 (119905) gt 119896119871mig(119905) = (119880119901119896 cpu(119905) minus 119896) lowast 119862119896 Also 119901119895 is the least loaded host in host list If 119880119901119895 (119905) ge 119896 all hostsare overloaded and this algorithm does not perform migration else if 119880119901119895 (119905) lt 119896MMAsends command to 119901119896 that 119901119895 is its migration destination host(2) Then min119871migrate(119905) (119896 minus 119880119901119895 (119905)) lowast 119862119895 is the largest accepted migration load The largestaccepted VM number is calculated according to 119899mig
119895(119905) = floor(min119871mig(119905) (119896 minus 119880119901119895(119905)) lowast 119862119895(119871V119894119896 (119905)))
(3) Calculate the communication cost CommV119894119896 119901119895 (119905) between every VM of 119901119896 and host 119901119895 and thesum of communication cost CommV119894119896 119901119896minusV119894119896 (119905) between the VM in 119901119896 and the rest VMs in 119901119896 TheVM which has min119894(CommV119894119896 119901119896minusV119894119896 (119905) minus CommV119894119896 119901119895 (119905)) is selected into the VM set 119904119896119895(119905) Then theselected VM is removed from 119901119896 while 119901119895 adds the selected VM Accordingly the communicationrelations of VMsrsquo communication are updated(4) If the number of VMs in 119904119896119895(119905) is less than 119899
mig119895(119905) back to Step 3 Otherwise output its planned
migration set 119904119896119895(119905) of 119901119896(5) If 119871 119904119896119895(119905) le 119871mig(119905) VMM of host 119901119896 and 119901119895 starts the migration
Algorithm 1 Communication cost based VM dynamic migration algorithm
4 Experiment Results and Analysis
41 Experiment Design To validate the effectiveness of theproposed VM based HLA simulation load balancing methodin CSim experiments have been designed and implementedThe simulations were run in a system comprising 2 nodesof Lenovo 8200t 2 nodes of HP 6300 Pro MT 6 nodes ofHP Compaq 8000 Elite CMT and a 100Mbitsec Ethernetconnection among all the nodes The node of Lenovo 8200thad an Intel i7-870 (8 cores) 293GHz CPU and 8G MEMThe node of HP 6300 had an Intel i5-3470 (4 cores) 32 GHzCPU and 4G MEM The node of Compaq 8000 had an IntelCore 2 E8400 (2 cores) 300GHz CPU and 2G MEM
The nodes run a paravirtualized Linux CentOS 56 kernelas a privileged virtual machine on top of the Xen hypervisor401 [30] The guest virtual machines are configured tosingle core and run the same version of the Linux kernel asthat of the privileged one HLA platform was AST-RTI [4950] version 20 performing communication through TCPIPconnections
Moreover as our benchmark a practical HLA armoredforce game for tactical training was developed The gamecoded in CC++ was used to conduct experiments and ana-lyze the performance of our approach The scenario for ourexperiments was a simulation of battle engagement game ofred and blue tank forces which were hierarchically organizedas Platoon (P) Company (C) Battalion (B) and Regiment(R) The tank effectuated random selection of several tacticalroutes and engagement strategies in two-dimensional spacethat was within range of some military training location
The organization structure of tank forces is illustratedin Figure 6 which shows that red forces are formed hierar-chically in 3 to 3 organization This means that every redcompany has 3 platoons and every platoon has 3 tanks whilefor the blue side it is formed in 4 to 4 organization which
Table 1 The number of VMs in different game scenarios
Scenario (or scale) Numberof VMs
1 red Company versus 1 blue Company 91 red Battalion versus 1 blue Company 181 red Company versus 1 blue Battalion 251 red Battalion versus 1 blue Battalion 341 red Battalion + 1 red Company versus 1 blue Battalion 381 red Battalion + 1 red Company versus 1 blue Battalion+ 1 blue Company 43
1 red Battalion + 2 red Company versus 1 blue Battalion+ 1 blue company 47
1 red Battalion + 2 red Company versus 1 blue Battalion+ 2 blue Company 52
2 red Battalion versus 1 blue Battalion + 2 blueCompany 57
means every blue company has 4 platoons and every platoonhas 4 tanks
In order to accomplish such simulations we cluster tankentities into VMs according to their military affiliations Theabbreviations are P Platoon C Company B Battalion RRegiment r red b blue
Table 1 shows the experimentsrsquo deployment Each VMcontains one federate in the experiments because computa-tion and communication costs are mainly due to the numberof tank entities When the number of entities in one VM isfixed the number of federates has little impact on the VMrsquoscosts as interhost communication cost is normally muchgreater than local host cost
Moreover each red tank Company is deployed with4VMs which are Platoon-1 (P-1) P-2 P-3 and Company
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
Figure 1 The mapping relations of entity federate VM and host
Federate 1
Federate 2
Federate 3
Federate N
F 1
F 2
F NEntities
From entities to federateto VM and to host VM migration strategy
VM
VM
VM
VMM
VM VM
HostHost
Host
Host
Host
Host
VM
VM
MMA
Migration command
Figure 2 The strategy of clustering entities to federate to VM and to host
(i) An entity often refers to a simulation object such as atank an airplane and a helicopter or a car
(ii) A federate is the encapsulated HLA form of one ormore entities
(iii) VM is the container of one or more federates becauseVM is the basic service unit of userrsquos request in CSim
(iv) Host is the container of VMs which means one ormore VM could be consolidated to one host mostlydepending on the hostrsquos computation ability
As previously mentioned federate migration is enabledby VM in this paper The mapping relations among the fourconcepts are illustrated in Figure 1
The primary clustering strategy of multiscale HLA simu-lation implementation is shown in Figure 2 where VM basedmigration algorithm and implementation approach will beillustrated
In the following sections we study dynamic VM migra-tion mechanisms Section 31 proposes federatesrsquo distributionand communication architecture Section 32 develops thealgorithm solving the problem when to migrate VMs whatVMs are to be migrated and where the migrating VMs are tobe contained
31 VM Distribution and Communication Architecture InHLA simulations entities are often encapsulated in federateswhich are then located into VMs deployed in differenthosts Each federate executes the simulation by messagecommunication The federate communication architecture isshown in Figure 3
In this architecture each VM encapsulates a local-RTIcomponent (LRC) and there is one server-RTI running on ahost The RTI communication here is hybrid which meansthat HLA global management services including time syn-chronizations are executed by the communications amongserver-RTI and LRC Meanwhile federate to federate com-munications including object instance attribute reflectionsand interaction instance sendingreceiving are executed viaLRC in peer-to-peer mode [44]
The VMFederate distribution architecture is designed asFigure 4 each host is equipped with a VM monitor (VMM)in charge of monitoring the VMsrsquo load and hostsrsquo load In ourCSim every VMM is both a load monitor and a migrationexecutor VMM keeps track of running state of local hostand VMs Also VMM captures the snapshots of local VMsperiodically and then keeps them in a database located inlocal host
Meanwhile one host has a migration management agent(MMA) in charge of monitoring all VMsrsquo periodic statustriggering VM migration procedure when a host is over-loaded selecting the migrated federate sets and target VMsand sending the commandMMAmonitors all VMsrsquo periodicstatus by collecting status data from VMM of each host Thisis a pull manner because VMM pulls data from distributedhosts to MMA The frequency of this monitoring is set as 1sin this paper
32 VM Migration Algorithm Based on the architectureshown in Figures 3 and 4 this section delivers a solutionwhich handles dynamic load imbalance in HLA federationsconsidering both computational and communication costs
Mathematical Problems in Engineering 5
VM VM
VM
Global management Global management
Federate to federatecommunication
Global management
Federate to federate
communication
Federate tofederate
communication
Server-RTI
LRC local RTI component
LRC LRC
LRC
Figure 3 VMFederate communication architecture
VM
Host Host
VM
Host
VM
VM
VMM VMM VMM
MigrationcommandMigration
command
VM snapshots
State monitoring
Periodic sampling
MMA migration mgmt agentLRC local RTI component
Server-RTI
MMALRC
LRC
LRC
LRC
VM and host states
middot middot middot
Figure 4 VMFederate distribution architecture
Let us start from analyzing the utilization and workload ofhosts
321 Hostsrsquo Utilization Threshold 119880119901119895(119905) is the utilization ofhost 119901119895 at time 119905
where 119880cpu(119905) is CPU utilization of host 119901119895 (in percent) and119880mem(119905) is memory utilization of host 119901119895 (in percent) 120572 isa coefficient representing the relative importance betweenCPU utilization and memory utilization As both CPU andmemory are equally important for running VM 120572 is set to05
The utilization threshold host 119901119895 is Th119901119895 or 119896 which isa parameter that allows the adjustment of the effect of themethod the lower 119896 is the higher the possible overload is andthe higher possibility of migration is For the determinationof utilization threshold the following equation is used 119896 =lceil(1 + 120573) sdot ((sum
119899
119895=1119880119901119895)119899)rceil where 119896 is the CPU utilization
threshold for all hosts 119880119901119895 is load of node 119895 119899 is the numberof hosts in the simulation system and 0 le 120573 le 02 is anormalized constant To generate moderate migration herewe set 120573 = 01
Obviously there are two load states for the hosts
(i) If 119880119901119895(119905) ge 119896 host 119901119895 is overloaded
(ii) If 119880119901119895(119905) lt 119896 host 119901119895 is not overloaded
6 Mathematical Problems in Engineering
322 Load of Host and VM At time 119905 load 119871119901119895(119905) of host 119901119895is computed by the following equation
where 119862119895 is the computation capacity of host 119901119895 whichis the frequency of the hostrsquos CPU mapped onto millionsinstructions per second (MIPS) ratings of each core [45]
In this paper we assume that each federate has variableworkload throughout a simulation but we can use prelim-inary experiments to test the maximum loaded VMs of ahost according to their configurations Suppose that all VMsare homogeneously configured and have the same amount ofentities the load 119871V119894119895(119905) (MIPS) of VM V119894119895 at time 119905 is definedas follows
where 119899119895 is the number of VMs in host 119901119895 and the assumptionhere also enables that we canmigrate a set of VMs at one time(see algorithm studied later)
323 Communication Cost In CSim HLA federation fed-erates communicate with each other through the interac-tion class instance and object class instance As discussedin Section 22 we assume the interaction class instance isstochastically sent and object class instance is periodicallyupdated every time step for the correctness of simulationsThe communication bandwidth request (bits) between fed-erate 1198861 and 1198862 is as follows
Comm1198861 1198862 =1
sim steplowast obj ins bytes lowast 8 (4)
where obj ins bytes is the amount of object class instancesbytes exchanged every time step This means the communi-cation cost shown in Figure 5 is the object class instancesrsquorequirements of network bandwidth
Thenwe try to compute the communication cost betweenhost and VM and host and host Figure 5 illustrates theinteractions
In most cases a prerequisite is that VMs in a local hostcan communicate much faster than VMs among differenthostsTherefore wemust consider the two cases separately InFigure 5(a) the solid lines are communications among VMswithin hosts and the dashed lines are the communicationsamong VMs of different hosts After communication merg-ing we can get Figure 5(b) where the c1s represents the sumof communication costs between vm1 and host 119901119895
Therefore the communication cost at time 119905 between theVM 119886119894119896 and the host 119901119895 is defined as follows
With respect to two objectives of dynamic load balancingreducing the load of the overloaded hosts and decreasingthe interhost communication cost a dynamic load balancingmodel is proposed as follows
min 119911 (119905) =119899
sum
119895=1
Comm119901119896119901119895 (119905) st 119880119901119895 (119905) lt Th119901119895 (7)
The objective function 119911(119905) is to minimize the interhostcommunications between host 119901119896 with VM to be migratedand other hosts and the constraint is that each hostrsquoscomputation load is below its threshold
324 Migration Algorithm The migration model above is aNP-hard problem Many researchers have used heuristics tofind the optimal solutions and our approach is influencedby them including the works in [17 26 35 36 39ndash4345ndash48] However compared with existing researches ouralgorithm not only considers the periodical HLA object classcommunication cost but also migrates a set of VMs everytime decreasing the migration procedure latency comparedto most one-by-one federate migration methods
Suppose the overloaded host is 119901119896 and the destinationhost selected by migration management agent (MMA) is 119901119895which has least utilization in the simulation The heuristic isto select a set of VMs from 119901119896 to migrate to 119901119895 in order toreduce the load of 119901119896 and minimize the communication costafter migration The algorithm is illustrated in Algorithm 1
For the proposed algorithm in Algorithm 1 the timecomplexity of Steps 1ndash4 is119874(1198732ave) (119873ave is the average numberof VMs per host) and the time complexity of Step 5 is 119874(119898)Thus the time complexity is 119874(119898 lowast 1198732ave) Moreover thealgorithm is executed with the same frequency of MMAmonitoring all VMsrsquo periodic status that is 1s
Mathematical Problems in Engineering 7
Input VM list withm VMs host list with n hostsOutput Deployment that VM to host (VM119894 host119895 | 119894 isin (1 119898) 119895 isin (1 119899))Algorithm
(1) At time 119905 MMA finds that host 119901119896 is overloaded and needs VMmigration where 119880119901119896 (119905) gt 119896119871mig(119905) = (119880119901119896 cpu(119905) minus 119896) lowast 119862119896 Also 119901119895 is the least loaded host in host list If 119880119901119895 (119905) ge 119896 all hostsare overloaded and this algorithm does not perform migration else if 119880119901119895 (119905) lt 119896MMAsends command to 119901119896 that 119901119895 is its migration destination host(2) Then min119871migrate(119905) (119896 minus 119880119901119895 (119905)) lowast 119862119895 is the largest accepted migration load The largestaccepted VM number is calculated according to 119899mig
119895(119905) = floor(min119871mig(119905) (119896 minus 119880119901119895(119905)) lowast 119862119895(119871V119894119896 (119905)))
(3) Calculate the communication cost CommV119894119896 119901119895 (119905) between every VM of 119901119896 and host 119901119895 and thesum of communication cost CommV119894119896 119901119896minusV119894119896 (119905) between the VM in 119901119896 and the rest VMs in 119901119896 TheVM which has min119894(CommV119894119896 119901119896minusV119894119896 (119905) minus CommV119894119896 119901119895 (119905)) is selected into the VM set 119904119896119895(119905) Then theselected VM is removed from 119901119896 while 119901119895 adds the selected VM Accordingly the communicationrelations of VMsrsquo communication are updated(4) If the number of VMs in 119904119896119895(119905) is less than 119899
mig119895(119905) back to Step 3 Otherwise output its planned
migration set 119904119896119895(119905) of 119901119896(5) If 119871 119904119896119895(119905) le 119871mig(119905) VMM of host 119901119896 and 119901119895 starts the migration
Algorithm 1 Communication cost based VM dynamic migration algorithm
4 Experiment Results and Analysis
41 Experiment Design To validate the effectiveness of theproposed VM based HLA simulation load balancing methodin CSim experiments have been designed and implementedThe simulations were run in a system comprising 2 nodesof Lenovo 8200t 2 nodes of HP 6300 Pro MT 6 nodes ofHP Compaq 8000 Elite CMT and a 100Mbitsec Ethernetconnection among all the nodes The node of Lenovo 8200thad an Intel i7-870 (8 cores) 293GHz CPU and 8G MEMThe node of HP 6300 had an Intel i5-3470 (4 cores) 32 GHzCPU and 4G MEM The node of Compaq 8000 had an IntelCore 2 E8400 (2 cores) 300GHz CPU and 2G MEM
The nodes run a paravirtualized Linux CentOS 56 kernelas a privileged virtual machine on top of the Xen hypervisor401 [30] The guest virtual machines are configured tosingle core and run the same version of the Linux kernel asthat of the privileged one HLA platform was AST-RTI [4950] version 20 performing communication through TCPIPconnections
Moreover as our benchmark a practical HLA armoredforce game for tactical training was developed The gamecoded in CC++ was used to conduct experiments and ana-lyze the performance of our approach The scenario for ourexperiments was a simulation of battle engagement game ofred and blue tank forces which were hierarchically organizedas Platoon (P) Company (C) Battalion (B) and Regiment(R) The tank effectuated random selection of several tacticalroutes and engagement strategies in two-dimensional spacethat was within range of some military training location
The organization structure of tank forces is illustratedin Figure 6 which shows that red forces are formed hierar-chically in 3 to 3 organization This means that every redcompany has 3 platoons and every platoon has 3 tanks whilefor the blue side it is formed in 4 to 4 organization which
Table 1 The number of VMs in different game scenarios
Scenario (or scale) Numberof VMs
1 red Company versus 1 blue Company 91 red Battalion versus 1 blue Company 181 red Company versus 1 blue Battalion 251 red Battalion versus 1 blue Battalion 341 red Battalion + 1 red Company versus 1 blue Battalion 381 red Battalion + 1 red Company versus 1 blue Battalion+ 1 blue Company 43
1 red Battalion + 2 red Company versus 1 blue Battalion+ 1 blue company 47
1 red Battalion + 2 red Company versus 1 blue Battalion+ 2 blue Company 52
2 red Battalion versus 1 blue Battalion + 2 blueCompany 57
means every blue company has 4 platoons and every platoonhas 4 tanks
In order to accomplish such simulations we cluster tankentities into VMs according to their military affiliations Theabbreviations are P Platoon C Company B Battalion RRegiment r red b blue
Table 1 shows the experimentsrsquo deployment Each VMcontains one federate in the experiments because computa-tion and communication costs are mainly due to the numberof tank entities When the number of entities in one VM isfixed the number of federates has little impact on the VMrsquoscosts as interhost communication cost is normally muchgreater than local host cost
Moreover each red tank Company is deployed with4VMs which are Platoon-1 (P-1) P-2 P-3 and Company
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
where 119880cpu(119905) is CPU utilization of host 119901119895 (in percent) and119880mem(119905) is memory utilization of host 119901119895 (in percent) 120572 isa coefficient representing the relative importance betweenCPU utilization and memory utilization As both CPU andmemory are equally important for running VM 120572 is set to05
The utilization threshold host 119901119895 is Th119901119895 or 119896 which isa parameter that allows the adjustment of the effect of themethod the lower 119896 is the higher the possible overload is andthe higher possibility of migration is For the determinationof utilization threshold the following equation is used 119896 =lceil(1 + 120573) sdot ((sum
119899
119895=1119880119901119895)119899)rceil where 119896 is the CPU utilization
threshold for all hosts 119880119901119895 is load of node 119895 119899 is the numberof hosts in the simulation system and 0 le 120573 le 02 is anormalized constant To generate moderate migration herewe set 120573 = 01
Obviously there are two load states for the hosts
(i) If 119880119901119895(119905) ge 119896 host 119901119895 is overloaded
(ii) If 119880119901119895(119905) lt 119896 host 119901119895 is not overloaded
6 Mathematical Problems in Engineering
322 Load of Host and VM At time 119905 load 119871119901119895(119905) of host 119901119895is computed by the following equation
where 119862119895 is the computation capacity of host 119901119895 whichis the frequency of the hostrsquos CPU mapped onto millionsinstructions per second (MIPS) ratings of each core [45]
In this paper we assume that each federate has variableworkload throughout a simulation but we can use prelim-inary experiments to test the maximum loaded VMs of ahost according to their configurations Suppose that all VMsare homogeneously configured and have the same amount ofentities the load 119871V119894119895(119905) (MIPS) of VM V119894119895 at time 119905 is definedas follows
where 119899119895 is the number of VMs in host 119901119895 and the assumptionhere also enables that we canmigrate a set of VMs at one time(see algorithm studied later)
323 Communication Cost In CSim HLA federation fed-erates communicate with each other through the interac-tion class instance and object class instance As discussedin Section 22 we assume the interaction class instance isstochastically sent and object class instance is periodicallyupdated every time step for the correctness of simulationsThe communication bandwidth request (bits) between fed-erate 1198861 and 1198862 is as follows
Comm1198861 1198862 =1
sim steplowast obj ins bytes lowast 8 (4)
where obj ins bytes is the amount of object class instancesbytes exchanged every time step This means the communi-cation cost shown in Figure 5 is the object class instancesrsquorequirements of network bandwidth
Thenwe try to compute the communication cost betweenhost and VM and host and host Figure 5 illustrates theinteractions
In most cases a prerequisite is that VMs in a local hostcan communicate much faster than VMs among differenthostsTherefore wemust consider the two cases separately InFigure 5(a) the solid lines are communications among VMswithin hosts and the dashed lines are the communicationsamong VMs of different hosts After communication merg-ing we can get Figure 5(b) where the c1s represents the sumof communication costs between vm1 and host 119901119895
Therefore the communication cost at time 119905 between theVM 119886119894119896 and the host 119901119895 is defined as follows
With respect to two objectives of dynamic load balancingreducing the load of the overloaded hosts and decreasingthe interhost communication cost a dynamic load balancingmodel is proposed as follows
min 119911 (119905) =119899
sum
119895=1
Comm119901119896119901119895 (119905) st 119880119901119895 (119905) lt Th119901119895 (7)
The objective function 119911(119905) is to minimize the interhostcommunications between host 119901119896 with VM to be migratedand other hosts and the constraint is that each hostrsquoscomputation load is below its threshold
324 Migration Algorithm The migration model above is aNP-hard problem Many researchers have used heuristics tofind the optimal solutions and our approach is influencedby them including the works in [17 26 35 36 39ndash4345ndash48] However compared with existing researches ouralgorithm not only considers the periodical HLA object classcommunication cost but also migrates a set of VMs everytime decreasing the migration procedure latency comparedto most one-by-one federate migration methods
Suppose the overloaded host is 119901119896 and the destinationhost selected by migration management agent (MMA) is 119901119895which has least utilization in the simulation The heuristic isto select a set of VMs from 119901119896 to migrate to 119901119895 in order toreduce the load of 119901119896 and minimize the communication costafter migration The algorithm is illustrated in Algorithm 1
For the proposed algorithm in Algorithm 1 the timecomplexity of Steps 1ndash4 is119874(1198732ave) (119873ave is the average numberof VMs per host) and the time complexity of Step 5 is 119874(119898)Thus the time complexity is 119874(119898 lowast 1198732ave) Moreover thealgorithm is executed with the same frequency of MMAmonitoring all VMsrsquo periodic status that is 1s
Mathematical Problems in Engineering 7
Input VM list withm VMs host list with n hostsOutput Deployment that VM to host (VM119894 host119895 | 119894 isin (1 119898) 119895 isin (1 119899))Algorithm
(1) At time 119905 MMA finds that host 119901119896 is overloaded and needs VMmigration where 119880119901119896 (119905) gt 119896119871mig(119905) = (119880119901119896 cpu(119905) minus 119896) lowast 119862119896 Also 119901119895 is the least loaded host in host list If 119880119901119895 (119905) ge 119896 all hostsare overloaded and this algorithm does not perform migration else if 119880119901119895 (119905) lt 119896MMAsends command to 119901119896 that 119901119895 is its migration destination host(2) Then min119871migrate(119905) (119896 minus 119880119901119895 (119905)) lowast 119862119895 is the largest accepted migration load The largestaccepted VM number is calculated according to 119899mig
119895(119905) = floor(min119871mig(119905) (119896 minus 119880119901119895(119905)) lowast 119862119895(119871V119894119896 (119905)))
(3) Calculate the communication cost CommV119894119896 119901119895 (119905) between every VM of 119901119896 and host 119901119895 and thesum of communication cost CommV119894119896 119901119896minusV119894119896 (119905) between the VM in 119901119896 and the rest VMs in 119901119896 TheVM which has min119894(CommV119894119896 119901119896minusV119894119896 (119905) minus CommV119894119896 119901119895 (119905)) is selected into the VM set 119904119896119895(119905) Then theselected VM is removed from 119901119896 while 119901119895 adds the selected VM Accordingly the communicationrelations of VMsrsquo communication are updated(4) If the number of VMs in 119904119896119895(119905) is less than 119899
mig119895(119905) back to Step 3 Otherwise output its planned
migration set 119904119896119895(119905) of 119901119896(5) If 119871 119904119896119895(119905) le 119871mig(119905) VMM of host 119901119896 and 119901119895 starts the migration
Algorithm 1 Communication cost based VM dynamic migration algorithm
4 Experiment Results and Analysis
41 Experiment Design To validate the effectiveness of theproposed VM based HLA simulation load balancing methodin CSim experiments have been designed and implementedThe simulations were run in a system comprising 2 nodesof Lenovo 8200t 2 nodes of HP 6300 Pro MT 6 nodes ofHP Compaq 8000 Elite CMT and a 100Mbitsec Ethernetconnection among all the nodes The node of Lenovo 8200thad an Intel i7-870 (8 cores) 293GHz CPU and 8G MEMThe node of HP 6300 had an Intel i5-3470 (4 cores) 32 GHzCPU and 4G MEM The node of Compaq 8000 had an IntelCore 2 E8400 (2 cores) 300GHz CPU and 2G MEM
The nodes run a paravirtualized Linux CentOS 56 kernelas a privileged virtual machine on top of the Xen hypervisor401 [30] The guest virtual machines are configured tosingle core and run the same version of the Linux kernel asthat of the privileged one HLA platform was AST-RTI [4950] version 20 performing communication through TCPIPconnections
Moreover as our benchmark a practical HLA armoredforce game for tactical training was developed The gamecoded in CC++ was used to conduct experiments and ana-lyze the performance of our approach The scenario for ourexperiments was a simulation of battle engagement game ofred and blue tank forces which were hierarchically organizedas Platoon (P) Company (C) Battalion (B) and Regiment(R) The tank effectuated random selection of several tacticalroutes and engagement strategies in two-dimensional spacethat was within range of some military training location
The organization structure of tank forces is illustratedin Figure 6 which shows that red forces are formed hierar-chically in 3 to 3 organization This means that every redcompany has 3 platoons and every platoon has 3 tanks whilefor the blue side it is formed in 4 to 4 organization which
Table 1 The number of VMs in different game scenarios
Scenario (or scale) Numberof VMs
1 red Company versus 1 blue Company 91 red Battalion versus 1 blue Company 181 red Company versus 1 blue Battalion 251 red Battalion versus 1 blue Battalion 341 red Battalion + 1 red Company versus 1 blue Battalion 381 red Battalion + 1 red Company versus 1 blue Battalion+ 1 blue Company 43
1 red Battalion + 2 red Company versus 1 blue Battalion+ 1 blue company 47
1 red Battalion + 2 red Company versus 1 blue Battalion+ 2 blue Company 52
2 red Battalion versus 1 blue Battalion + 2 blueCompany 57
means every blue company has 4 platoons and every platoonhas 4 tanks
In order to accomplish such simulations we cluster tankentities into VMs according to their military affiliations Theabbreviations are P Platoon C Company B Battalion RRegiment r red b blue
Table 1 shows the experimentsrsquo deployment Each VMcontains one federate in the experiments because computa-tion and communication costs are mainly due to the numberof tank entities When the number of entities in one VM isfixed the number of federates has little impact on the VMrsquoscosts as interhost communication cost is normally muchgreater than local host cost
Moreover each red tank Company is deployed with4VMs which are Platoon-1 (P-1) P-2 P-3 and Company
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
where 119862119895 is the computation capacity of host 119901119895 whichis the frequency of the hostrsquos CPU mapped onto millionsinstructions per second (MIPS) ratings of each core [45]
In this paper we assume that each federate has variableworkload throughout a simulation but we can use prelim-inary experiments to test the maximum loaded VMs of ahost according to their configurations Suppose that all VMsare homogeneously configured and have the same amount ofentities the load 119871V119894119895(119905) (MIPS) of VM V119894119895 at time 119905 is definedas follows
where 119899119895 is the number of VMs in host 119901119895 and the assumptionhere also enables that we canmigrate a set of VMs at one time(see algorithm studied later)
323 Communication Cost In CSim HLA federation fed-erates communicate with each other through the interac-tion class instance and object class instance As discussedin Section 22 we assume the interaction class instance isstochastically sent and object class instance is periodicallyupdated every time step for the correctness of simulationsThe communication bandwidth request (bits) between fed-erate 1198861 and 1198862 is as follows
Comm1198861 1198862 =1
sim steplowast obj ins bytes lowast 8 (4)
where obj ins bytes is the amount of object class instancesbytes exchanged every time step This means the communi-cation cost shown in Figure 5 is the object class instancesrsquorequirements of network bandwidth
Thenwe try to compute the communication cost betweenhost and VM and host and host Figure 5 illustrates theinteractions
In most cases a prerequisite is that VMs in a local hostcan communicate much faster than VMs among differenthostsTherefore wemust consider the two cases separately InFigure 5(a) the solid lines are communications among VMswithin hosts and the dashed lines are the communicationsamong VMs of different hosts After communication merg-ing we can get Figure 5(b) where the c1s represents the sumof communication costs between vm1 and host 119901119895
Therefore the communication cost at time 119905 between theVM 119886119894119896 and the host 119901119895 is defined as follows
With respect to two objectives of dynamic load balancingreducing the load of the overloaded hosts and decreasingthe interhost communication cost a dynamic load balancingmodel is proposed as follows
min 119911 (119905) =119899
sum
119895=1
Comm119901119896119901119895 (119905) st 119880119901119895 (119905) lt Th119901119895 (7)
The objective function 119911(119905) is to minimize the interhostcommunications between host 119901119896 with VM to be migratedand other hosts and the constraint is that each hostrsquoscomputation load is below its threshold
324 Migration Algorithm The migration model above is aNP-hard problem Many researchers have used heuristics tofind the optimal solutions and our approach is influencedby them including the works in [17 26 35 36 39ndash4345ndash48] However compared with existing researches ouralgorithm not only considers the periodical HLA object classcommunication cost but also migrates a set of VMs everytime decreasing the migration procedure latency comparedto most one-by-one federate migration methods
Suppose the overloaded host is 119901119896 and the destinationhost selected by migration management agent (MMA) is 119901119895which has least utilization in the simulation The heuristic isto select a set of VMs from 119901119896 to migrate to 119901119895 in order toreduce the load of 119901119896 and minimize the communication costafter migration The algorithm is illustrated in Algorithm 1
For the proposed algorithm in Algorithm 1 the timecomplexity of Steps 1ndash4 is119874(1198732ave) (119873ave is the average numberof VMs per host) and the time complexity of Step 5 is 119874(119898)Thus the time complexity is 119874(119898 lowast 1198732ave) Moreover thealgorithm is executed with the same frequency of MMAmonitoring all VMsrsquo periodic status that is 1s
Mathematical Problems in Engineering 7
Input VM list withm VMs host list with n hostsOutput Deployment that VM to host (VM119894 host119895 | 119894 isin (1 119898) 119895 isin (1 119899))Algorithm
(1) At time 119905 MMA finds that host 119901119896 is overloaded and needs VMmigration where 119880119901119896 (119905) gt 119896119871mig(119905) = (119880119901119896 cpu(119905) minus 119896) lowast 119862119896 Also 119901119895 is the least loaded host in host list If 119880119901119895 (119905) ge 119896 all hostsare overloaded and this algorithm does not perform migration else if 119880119901119895 (119905) lt 119896MMAsends command to 119901119896 that 119901119895 is its migration destination host(2) Then min119871migrate(119905) (119896 minus 119880119901119895 (119905)) lowast 119862119895 is the largest accepted migration load The largestaccepted VM number is calculated according to 119899mig
119895(119905) = floor(min119871mig(119905) (119896 minus 119880119901119895(119905)) lowast 119862119895(119871V119894119896 (119905)))
(3) Calculate the communication cost CommV119894119896 119901119895 (119905) between every VM of 119901119896 and host 119901119895 and thesum of communication cost CommV119894119896 119901119896minusV119894119896 (119905) between the VM in 119901119896 and the rest VMs in 119901119896 TheVM which has min119894(CommV119894119896 119901119896minusV119894119896 (119905) minus CommV119894119896 119901119895 (119905)) is selected into the VM set 119904119896119895(119905) Then theselected VM is removed from 119901119896 while 119901119895 adds the selected VM Accordingly the communicationrelations of VMsrsquo communication are updated(4) If the number of VMs in 119904119896119895(119905) is less than 119899
mig119895(119905) back to Step 3 Otherwise output its planned
migration set 119904119896119895(119905) of 119901119896(5) If 119871 119904119896119895(119905) le 119871mig(119905) VMM of host 119901119896 and 119901119895 starts the migration
Algorithm 1 Communication cost based VM dynamic migration algorithm
4 Experiment Results and Analysis
41 Experiment Design To validate the effectiveness of theproposed VM based HLA simulation load balancing methodin CSim experiments have been designed and implementedThe simulations were run in a system comprising 2 nodesof Lenovo 8200t 2 nodes of HP 6300 Pro MT 6 nodes ofHP Compaq 8000 Elite CMT and a 100Mbitsec Ethernetconnection among all the nodes The node of Lenovo 8200thad an Intel i7-870 (8 cores) 293GHz CPU and 8G MEMThe node of HP 6300 had an Intel i5-3470 (4 cores) 32 GHzCPU and 4G MEM The node of Compaq 8000 had an IntelCore 2 E8400 (2 cores) 300GHz CPU and 2G MEM
The nodes run a paravirtualized Linux CentOS 56 kernelas a privileged virtual machine on top of the Xen hypervisor401 [30] The guest virtual machines are configured tosingle core and run the same version of the Linux kernel asthat of the privileged one HLA platform was AST-RTI [4950] version 20 performing communication through TCPIPconnections
Moreover as our benchmark a practical HLA armoredforce game for tactical training was developed The gamecoded in CC++ was used to conduct experiments and ana-lyze the performance of our approach The scenario for ourexperiments was a simulation of battle engagement game ofred and blue tank forces which were hierarchically organizedas Platoon (P) Company (C) Battalion (B) and Regiment(R) The tank effectuated random selection of several tacticalroutes and engagement strategies in two-dimensional spacethat was within range of some military training location
The organization structure of tank forces is illustratedin Figure 6 which shows that red forces are formed hierar-chically in 3 to 3 organization This means that every redcompany has 3 platoons and every platoon has 3 tanks whilefor the blue side it is formed in 4 to 4 organization which
Table 1 The number of VMs in different game scenarios
Scenario (or scale) Numberof VMs
1 red Company versus 1 blue Company 91 red Battalion versus 1 blue Company 181 red Company versus 1 blue Battalion 251 red Battalion versus 1 blue Battalion 341 red Battalion + 1 red Company versus 1 blue Battalion 381 red Battalion + 1 red Company versus 1 blue Battalion+ 1 blue Company 43
1 red Battalion + 2 red Company versus 1 blue Battalion+ 1 blue company 47
1 red Battalion + 2 red Company versus 1 blue Battalion+ 2 blue Company 52
2 red Battalion versus 1 blue Battalion + 2 blueCompany 57
means every blue company has 4 platoons and every platoonhas 4 tanks
In order to accomplish such simulations we cluster tankentities into VMs according to their military affiliations Theabbreviations are P Platoon C Company B Battalion RRegiment r red b blue
Table 1 shows the experimentsrsquo deployment Each VMcontains one federate in the experiments because computa-tion and communication costs are mainly due to the numberof tank entities When the number of entities in one VM isfixed the number of federates has little impact on the VMrsquoscosts as interhost communication cost is normally muchgreater than local host cost
Moreover each red tank Company is deployed with4VMs which are Platoon-1 (P-1) P-2 P-3 and Company
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
Input VM list withm VMs host list with n hostsOutput Deployment that VM to host (VM119894 host119895 | 119894 isin (1 119898) 119895 isin (1 119899))Algorithm
(1) At time 119905 MMA finds that host 119901119896 is overloaded and needs VMmigration where 119880119901119896 (119905) gt 119896119871mig(119905) = (119880119901119896 cpu(119905) minus 119896) lowast 119862119896 Also 119901119895 is the least loaded host in host list If 119880119901119895 (119905) ge 119896 all hostsare overloaded and this algorithm does not perform migration else if 119880119901119895 (119905) lt 119896MMAsends command to 119901119896 that 119901119895 is its migration destination host(2) Then min119871migrate(119905) (119896 minus 119880119901119895 (119905)) lowast 119862119895 is the largest accepted migration load The largestaccepted VM number is calculated according to 119899mig
119895(119905) = floor(min119871mig(119905) (119896 minus 119880119901119895(119905)) lowast 119862119895(119871V119894119896 (119905)))
(3) Calculate the communication cost CommV119894119896 119901119895 (119905) between every VM of 119901119896 and host 119901119895 and thesum of communication cost CommV119894119896 119901119896minusV119894119896 (119905) between the VM in 119901119896 and the rest VMs in 119901119896 TheVM which has min119894(CommV119894119896 119901119896minusV119894119896 (119905) minus CommV119894119896 119901119895 (119905)) is selected into the VM set 119904119896119895(119905) Then theselected VM is removed from 119901119896 while 119901119895 adds the selected VM Accordingly the communicationrelations of VMsrsquo communication are updated(4) If the number of VMs in 119904119896119895(119905) is less than 119899
mig119895(119905) back to Step 3 Otherwise output its planned
migration set 119904119896119895(119905) of 119901119896(5) If 119871 119904119896119895(119905) le 119871mig(119905) VMM of host 119901119896 and 119901119895 starts the migration
Algorithm 1 Communication cost based VM dynamic migration algorithm
4 Experiment Results and Analysis
41 Experiment Design To validate the effectiveness of theproposed VM based HLA simulation load balancing methodin CSim experiments have been designed and implementedThe simulations were run in a system comprising 2 nodesof Lenovo 8200t 2 nodes of HP 6300 Pro MT 6 nodes ofHP Compaq 8000 Elite CMT and a 100Mbitsec Ethernetconnection among all the nodes The node of Lenovo 8200thad an Intel i7-870 (8 cores) 293GHz CPU and 8G MEMThe node of HP 6300 had an Intel i5-3470 (4 cores) 32 GHzCPU and 4G MEM The node of Compaq 8000 had an IntelCore 2 E8400 (2 cores) 300GHz CPU and 2G MEM
The nodes run a paravirtualized Linux CentOS 56 kernelas a privileged virtual machine on top of the Xen hypervisor401 [30] The guest virtual machines are configured tosingle core and run the same version of the Linux kernel asthat of the privileged one HLA platform was AST-RTI [4950] version 20 performing communication through TCPIPconnections
Moreover as our benchmark a practical HLA armoredforce game for tactical training was developed The gamecoded in CC++ was used to conduct experiments and ana-lyze the performance of our approach The scenario for ourexperiments was a simulation of battle engagement game ofred and blue tank forces which were hierarchically organizedas Platoon (P) Company (C) Battalion (B) and Regiment(R) The tank effectuated random selection of several tacticalroutes and engagement strategies in two-dimensional spacethat was within range of some military training location
The organization structure of tank forces is illustratedin Figure 6 which shows that red forces are formed hierar-chically in 3 to 3 organization This means that every redcompany has 3 platoons and every platoon has 3 tanks whilefor the blue side it is formed in 4 to 4 organization which
Table 1 The number of VMs in different game scenarios
Scenario (or scale) Numberof VMs
1 red Company versus 1 blue Company 91 red Battalion versus 1 blue Company 181 red Company versus 1 blue Battalion 251 red Battalion versus 1 blue Battalion 341 red Battalion + 1 red Company versus 1 blue Battalion 381 red Battalion + 1 red Company versus 1 blue Battalion+ 1 blue Company 43
1 red Battalion + 2 red Company versus 1 blue Battalion+ 1 blue company 47
1 red Battalion + 2 red Company versus 1 blue Battalion+ 2 blue Company 52
2 red Battalion versus 1 blue Battalion + 2 blueCompany 57
means every blue company has 4 platoons and every platoonhas 4 tanks
In order to accomplish such simulations we cluster tankentities into VMs according to their military affiliations Theabbreviations are P Platoon C Company B Battalion RRegiment r red b blue
Table 1 shows the experimentsrsquo deployment Each VMcontains one federate in the experiments because computa-tion and communication costs are mainly due to the numberof tank entities When the number of entities in one VM isfixed the number of federates has little impact on the VMrsquoscosts as interhost communication cost is normally muchgreater than local host cost
Moreover each red tank Company is deployed with4VMs which are Platoon-1 (P-1) P-2 P-3 and Company
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
Figure 6 Hierarchical organizational structure of red and blue forces
tank Each blue tankCompany is deployedwith 5VMswhichare Platoon-1 P-2 P-3 P-4 and Company tank
To fulfill migration algorithm addressed in Section 32the communication cost between VMFederates were esti-mated according to periodic HLA object class instanceswhile ignoring stochasticHLA interaction class instances Forexample rR1B1C1 needs to report its information by sendingits object class instances to rR1B1 every simulation step Itsobject class contains the information of ID name positionfuel consumption ammunition and so forth Accordingto this we can estimate the size of its object class forinstance 48 bytes Assuming the simulation step is 50msthen the communication cost caused by the object classinstance is 960 bytessec By using this method we can getthe communication cost among all the federates
42 Experimental Results and Analyses In order to evaluatethe proposed VM based migration algorithmrsquos efficiency theexperiments were accomplished in two test case groups overheterogeneous nondedicated sets of resources applying anincreasing large load to the distributed system In the first testcase group the effectiveness of the dynamic load balancingsystem was observed as distributed load imbalances occur Inthe second test case group to analyze the detection of externalbackground load an external load is added in the system andthe balancing reaction is observed
(1) Reactivity to Load Imbalances In this test case all thedistributed simulations were deployed based on an initialstatic partitioning that evenly placed the VMFederates onthe resources However due to the resource heterogeneitycharacteristics and variable federate loads the simulationshows an uneven distribution of load decreasing the simu-lationsrsquo performance In order to evaluate the balancing sys-temrsquos reaction to load imbalances and the VM encapsulationsimpact on simulation the balanced VM based simulationrsquosperformance was compared with static distribution wrappedand unwrapped with VM In this case of experiments thesystem comprehended the run of the experimental scenariowith a configuration of federates that ranged from 9 to 57 (seeTable 1)
To provide trustworthy results each execution time inour graphs represents the average of 20 runs For everymean value of simulation execution time a 95 confidenceinterval was evaluated The half-widths of all confidenceintervals are less than 5 of their respective mean valuesAccording to Figure 7 the proposed dynamic balancingalgorithm and VM migration improved the performance ofHLA-based simulations on large-scale distributed systemsin most of the experiments When the distributed load wasunder 20 federates the balancing schemersquos improvementis unnoticeable or nonexistent because the simulations didnot require any load balancing In this case the balancingjust caused a small overhead (21) for the distributedsystem consuming computing from the resource where theMMAwas deployed A noticeable improvement was detectedwith experiments over 25 federates because considerableload imbalances occurred during the simulation along withthe different deployment of VMs and the heterogeneity ofresources caused an imbalanced division of load Then ahigh increase in execution time in the balanced system isobserved when the number of federates is over 50 Thisincreases evidence that the distributed system is reachinga saturation point in which the balancing system cannotimprove the simulation performance since all resources arebecoming totally overloaded
In addition blue and red curves in Figure 7 show thatthe average overhead with VM encapsulation compared towithout VM in all runs is 328 which means using VMis acceptable because of two reasons Firstly using VM livemigration techniques saves lots of simulation programmersrsquoefforts in realization of federate migrations Secondly whenthe number of federates is less than 50 that is below thesaturated point of the system the average execution timesaved is 2225 compared to the static distribution runswithout VMs
(2) Detection of Background Load In order to measure theefficiency of the load balancing system in detecting andreacting to the background loads external jobs are generatedusing a tool called Stress [5] Stress is a workload generatorfor POSIX systems and allows for a configurable amount
Mathematical Problems in Engineering 9
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80
VM staticNo VM staticVM dynamic balancing
Number of federates
Exec
utio
n tim
e (s)
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
Figure 7 Dynamic balancing scheme versus a static distribution foran increasing scale of federates
of CPU and memory stress on the system In the test casethe federates were deployed evenly on the distributed nodesand Stress was placed on two nodes of HP Compaq 8000workstation The load was 1-CPU bound 1 IO bound andone memory allocator process
As shown in Figure 8 the curves are similar to those inFigure 7 except that introduction of an external load causedan addition of execution time for experiments which have nodynamic balancing scheme However the saturated point isearlier (changes from 52 to 47) because of the external loadimposed on the distributed system Thus the load balancingsystem presented a performance improvement detecting theexternal load and triggering redistribution of load only whenthe distributed system is not saturated
5 Conclusions and Future Work
The paper proposes a VM based federate migration schemefor HLA system load balancing on Cloud Simulation Plat-form Contribution of this work could be summed in twoaspects (i) it proposed to use VM as the container of federateThe overhead brought by VM is about 333 according topapers [1 33 34] (in our tests it is around 328) (ii)It devised an algorithm of HLA load balancing under theconstraints of both computational and communication costsThe experiment results show that the migration schemeeffectively improved the efficiency of the HLA system withthe prerequisite that the distributed system is not saturated
As a preliminary work in Cloud computing based HLAsystem this research has a lot of future work to do Firstlythe computing granularity is still a difficult problem because
Number of federates
0
1000
2000
3000
4000
5000
6000
0 10 20 30 40 50 60
VM staticNo VM staticVM dynamic balancing
Exec
utio
n tim
e (s)
Figure 8 Capacity of the dynamic balancing scheme in detectingbackground load for an increasing scale of federates
VM is actually a heavy container for current resources andif one VM contains only one federate the federate shouldinclude as many simulation entities as possible Howevera big federate containing many entities may not be flexibleto migrate for load balancing Therefore it is complex todesign an appropriate computing granularity and this shouldbe solved in the future Secondly migration algorithm shouldbe designed to bemore adapted toHLA systems In this paperwe devised an algorithm considering both computational andcommunication cost However the algorithm neglected thestochastic interaction classesrsquo characteristics which may beconsidered in an intelligent way to enhance the efficiency ofload balancing in HLA
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
The research in this paper was supported by Grants 61104057and 61473013 from the Natural Science Foundation of Chinaand funding of the Science andTechnology onComplex LandSystems Simulation Laboratory (63963) The authors thankthe reviewers for their comments
References
[1] X Liu Q He X Qiu B Chen and K Huang ldquoCloud-basedcomputer simulation Towards planting existing simulation
10 Mathematical Problems in Engineering
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
software into the cloudrdquo Simulation Modelling Practice andTheory vol 26 pp 135ndash150 2012
[2] B H Li X Chai and L Zhang ldquoNew advances of the researchon cloud simulationrdquo in Advanced Methods Techniques andApplications inModeling and Simulation vol 4 of Proceedings inInformation andCommunications Technology pp 144ndash163 2012
[3] S Jafer Q Liu and G Wainer ldquoSynchronization methods inparallel and distributed discrete-event simulationrdquo SimulationModelling Practice and Theory vol 30 pp 54ndash73 2013
[4] R Fujimoto A Malik and A Park ldquoParallel and distributedsimulation in the cloudrdquo SCS Modeling and Simulation Maga-zine pp 1ndash10 2010
[5] AW Malik A J Park and R M Fujimoto ldquoAn optimistic par-allel simulation protocol for cloud computing environmentsrdquoSCS MampS Magazine vol 4 2010
[6] A Javor and A Fur ldquoSimulation on the Web with distributedmodels and intelligent agentsrdquo Simulation vol 88 no 9 pp1080ndash1092 2012
[7] IEEE Std 15161-2010 IEEE Standard for Modeling and Simu-lation (MampS) High Level Architecture (HLA) Framework andRules Specification 2010
[8] IEEE Std 15162-2010 IEEE Standard for Modeling and Sim-ulation (MampS) High Level Architecture (HLA) Object ModelTemplate (OMT) Specification 2010
[9] IEEE Standard 15161-2010mdashIEEE Standard for Modeling andSimulation (MampS) High Level Architecture (HLA)mdashFederateInterface Specification 2010
[10] S Radio D Parsons and V Deneen MODSAF Overview andMODSAFHistory [EBOL] 2006 httpwwwaiaiedacuksimarpiSUOMODULESmodsafhtml
[11] B McEnany ldquoCCTT SAF functional analysisrdquo in Proceedings ofthe 4th Conference on Computer Generated Forces and Behav-ioral Representation Institute for Simulation andTraining 1994
[12] A J Courtemanche and R L Wittman Jr ldquoOneSAF a productline approach for a next-generation CGFrdquo in Proceedings of the11th Computer Generated Forces Conference IEEE ComputerSociety Press Orlando Fla USA 2002
[13] One Semi-Automated Forces (OneSAF) ldquoOperationalRequirements Document (ORD) Version 11[EBOL]rdquo 2000httpwwwonesafnetcommunity
[14] B H Li X Chai Y Di H Yu Z Du and X Peng ldquoResearchon service oriented simulation gridrdquo in Proceedings of the IEEEInternational Symposium on Autonomous Decentralized Systems(ISADS rsquo05) pp 7ndash14 April 2005
[15] I Foster C Kesselman J M Nick et alThe Physiology of GridAn Open Grid Services Architecture 2003
[16] S Tuecke K Czajkowski and I Foster Open Grid ServicesInfrastructure (OGSI) 2003 httpwwwggforgdocumentsGFD15pdf
[17] A Boukerche and R E de Grande ldquoDynamic load balancingusing grid services for HLA-based simulations on large-scaledistributed systemsrdquo in Proceedings of the 13th IEEEACM Sym-posium on Distributed Simulation and Real-Time Applications(DS-RT rsquo09) pp 175ndash183 October 2009
[18] Amazon AWS 2014 httpawsamazoncom[19] Google httpscloudgooglecom[20] Softlayer 2014 httpwwwsoftlayercomCloud[21] R N Rodrigo R Ranjan A Beloglazov C A F de Rose and
R Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resource
provisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2011
[22] Intel Corporation System Virtualization-Theory and Implemen-tation Tsinghua University Press Beijing China 2009
[23] D Ruest and N Ruest Virtualization A Beginnerrsquos GuideMcGraw-Hill NewYork NY USA 2009
[24] M Rosenblum and T Garfinkel ldquoVirtual machine monitorscurrent technology and future trendsrdquo Computer vol 38 no5 pp 39ndash47 2005
[25] C Clark K Fraser S Hand et al ldquoLive migration of virtualmachinesrdquo in Proceedings of the 2nd ACMUSENIX Symposiumon Networked Systems Design amp Implementation (NSDI rsquo05)vol 2 pp 273ndash286 USENIX Association Berkeley Calif USA2005
[26] R E DeGrandeDynamic load balancing schemes for large-scaleHLA-based simulations [PhD thesis] University of OttawaOntario Canada 2012
[27] VMware 2014 httpwwwvmwarecom[28] Opennebula 2014 httpopennebulaorg[29] Eucalyptus httpwwweucalyptuscom[30] Xen httpwwwxenprojectorg[31] C Clark K Fraser S Hand et al ldquoLive migration of virtual
machinesrdquo in Proceedings of the 2nd conference on SymposiumonNetworked Systems Designamp Implementation (NSDI rsquo05) vol2 pp 273ndash286 2005
[32] F Travostino P Daspit L Gommans et al ldquoSeamless livemigration of virtual machines over the MANWANrdquo FutureGeneration Computer Systems vol 22 no 8 pp 901ndash907 2006
[33] A Menon J R Santos Y Turner G J Janakiraman and WZwaenepoelDiagnosing Performance Overheads in the Xen Vir-tual Machine Environment-Network 2014 httpwwwusenixorgeventsvee05full papersp13-menonpdf
[34] G Diwaker and G R C Ludmila XenMon QoS Monitor-ing and Performance Profiling Tool httpwwwhplhpcomtechreports2005HPL-2005-187pdf 2014
[35] G Tan and K C Lim ldquoLoad distribution services in HLArdquoin Proceedings of the 8th IEEE International Symposium onDistributed Simulation and Real-Time Applications (DS-RT rsquo04)pp 133ndash141 October 2004
[36] G Tan A Persson and R Ayani ldquoMigration of HLA federatesrdquoin Proceedings of the Simulation Interoperability Workshop (SIWrsquo05) San Diego Calif USA 2005
[37] W H Tao Task management and scheduling methods for grid-computing-based simulation [PhD thesis] National Universityof Defense Technology 2005
[38] W Cai S J Turner and H Zhao ldquoA load management systemfor running HLA-based simulation over the gridrdquo in Proceed-ings of the 6th IEEE International Symposium on DistributedSimulation and Real Time Applications pp 7ndash14 Fort WorthTex USA 2002
[39] T Alam and Z Raza ldquoA dynamic load balancing strategy withadaptive threshold based approachrdquo in Proceedings of the 2ndIEEE International Conference on Parallel Distributed and GridComputing (PDGC rsquo12) pp 927ndash932 Solan India December2012
[40] J Xu and K Hwang ldquoHeuristic methods for dynamic loadbalancing in a message-passing supercomputerrdquo in Proceedingsof the ACMIEEE conference Supercomputing (Supercomputingrsquo90) pp 888ndash897 New York NY USA November 1990
Mathematical Problems in Engineering 11
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
[41] A Y Zomaya and Y-H Teh ldquoObservations on using geneticalgorithms for dynamic load-balancingrdquo IEEE Transactions onParallel and Distributed Systems vol 12 no 9 pp 899ndash911 2001
[42] S Jin and B Ren ldquoA novel distributed dynamic load balancingmechanismrdquo in Proceedings of the International Conference onInformation Technology Computer Engineering and Manage-ment Sciences (ICM rsquo11) pp 133ndash137 Nanjing China September2011
[43] A Boukerche and S K Das ldquoReducing null messages overheadthrough load balancing in conservative distributed simulationsystemsrdquo Journal of Parallel and Distributed Computing vol 64no 3 pp 330ndash334 2004
[44] M Eklof M Sparf F Moradi and R Ayani ldquoPeer-to-peer-based resource management in support of HLA-Based dis-tributed simulationsrdquo Simulation vol 80 no 4-5 pp 181ndash1902004
[45] A Beloglazov and R Buyya ldquoOptimal online deterministicalgorithms and adaptive heuristics for energy and performanceefficient dynamic consolidation of virtual machines in Clouddata centersrdquo Concurrency Computation Practice and Experi-ence vol 24 no 13 pp 1397ndash1420 2012
[46] Q Long J Lin and Z Sun ldquoAgent scheduling model foradaptive dynamic load balancing in agent-based distributedsimulationsrdquo Simulation Modelling Practice and Theory vol 19no 4 pp 1021ndash1034 2011
[47] N Rodrigo R Ranjan A Beloglazov C A F de Rose andR Buyya ldquoCloudSim a toolkit for modeling and simulationof cloud computing environments and evaluation of resourceprovisioning algorithmsrdquo Software Practice and Experience vol41 no 1 pp 23ndash50 2014
[48] A Murtazaev and S Oh ldquoSercon server consolidation algo-rithm using live migration of virtual machines for greencomputingrdquo IETE Technical Review vol 28 no 3 pp 212ndash2312011
[49] Y Xu M Yu and X Wang ldquoResearch and development onAST-RTIrdquo in Systems Modeling and Simulation Theory andApplications vol 3398 of Lecture Notes in Computer Science pp361ndash366 2005
[50] N Li X-Y Peng M-H Zhang M Wang and G-H GongldquoMultimedia communication over HLARTIrdquo Simulation Mod-elling Practice and Theory vol 14 no 2 pp 161ndash176 2006
