www.eubra-bigsea.eu | [email protected] |@bigsea_eubr 1 D3.1: QoS Monitoring System Architecture Author(s) Ignacio Blanquer (UPV), Guilherme Maluf (UFMG), Walter dos Santos Filho (UFMG) Status Draft/Review/Approval/Final Version v1.0 Date 2/05/2016 Dissemination Level X PU: Public PP: Restricted to other programme participants (including the Commission) RE: Restricted to a group specified by the consortium (including the Commission) CO: Confidential, only for members of the consortium (including the Commission) Abstract: Europe - Brazil Collaboration of BIG Data Scientific Research through Cloud-Centric Applications (EUBra-BIGSEA) is a medium-scale research project funded by the European Commission under the Cooperation Programme, and the Ministry of Science and Technology (MCT) of Brazil in the frame of the third European-Brazilian coordinated call. The document has been produced with the co-funding of the European Commission and the MCT. The purpose of this report on the QoS Monitoring System Architecture is to define the software components that will collect the execution data from the cloud architecture, as well as the main components that intervene in the full process of deployment, configuration, contextualization and execution. EUBra-BIGSEA is funded by the European Commission under the Cooperation Programme, Horizon 2020 grant agreement No 690116. Este projeto é resultante da 3a Chamada Coordenada BR-UE em Tecnologias da Informação e Comunicação (TIC), anunciada pelo Ministério de Ciência, Tecnologia e Inovação (MCTI)
Implementation of the comments fromreviewersandfinalcandidateversion
Copyright notice: This work is licensed under the Creative Commons CC-BY 4.0 license. To view a copy of this license, visithttps://creativecommons.org/licenses/by/4.0.
Disclaimer: The content of the document herein is the sole responsibility of the publishers and it does not necessarily represent the viewsexpressedbytheEuropeanCommissionoritsservices.
The Quality of Service (QoS) architecture is the computational core of the EUBra-BIGSEA platform. TheperformanceofdataanalyticsapplicationsrunningontheEUBRra-BIGSEAplatformareprofiledinadvance,soaQoSguaranteeisdefinedbasedontheperformancerequirements.Then,themonitoringservicewillcloselyfollowtheexecutionsoitcanreactandrequestforadditionalresourcesifneeded.Thiswillbefedback to the performance estimationmodel so the allocation of resources for the next execution can bemoreprecise.Thisway,proactiveelasticitycanadaptthesysteminadvancesothatnodrasticactionsareneededforguaranteeingtheQoSoftheapplications,meanwhile,reactiveelasticitycanworkasasecondactiontorespondtochangesthatwerenotadequatelyhandledbytheproactiveapproach.Userswillnotneed to specify theexpected resources, leveraging the resourceestimation system todecide,within theacceptableboundaries.
Theproject identifiesthreetypesofworkloads,persistent,periodicbatchand interactive jobs,whichwillbeservedbydifferentschedulers.PersistentjobswillbeservedbytheMarathonscheduler,periodicjobsbymeans of Chronos scheduler, and interactive jobs through interactive shells (e.g. spark shells). Thoseschedulerswill deploy frameworks thatwill embed the executable services and negotiate the resourceswithMesos.ThoserequestswillbeinterceptedtoguaranteetheavailabilityofresourcesinMesos(e.g.byaddingmoreresourcestothesystem).Therefore,atthelevelofthecloudservices,theprojectwillcapturetheframeworkrequestsandinteractwithMesos,aswellasdealwiththeprovisionandmanagementofbaseVMstopursueanefficientuseofresources.QoS cloud services will deal with the reactive elasticity of the Mesos cluster and the physical (virtual)resources that Mesos base on. Frameworks could request additional resources in case of applicationstarvationanddependingontheprofileof theapplication.Then, themonitoringsystemwill triggersuchactions.ThechoiceforthemonitoringsystemisMonasca.MonascaisayoungOpenStackprojectformonitoring.IthasbeenselectedforitssuitabilitytoEUBra-BIGSEAusecaseandtheopportunitiesthatcouldarisefromthecollaborationwithOpenStack.Monascawillmonitortheresources,servicesandapplications,triggeringactionsifrequired.
1.1 ScopeoftheDocumentThisdocumentdescribestheQoSMonitoringSystemArchitectureaswellasthesoftwarearchitectureofothercloud-servicerelatedcomponentsandtheirinteractions.ThedocumentfollowstheSoftwareDesignSpecification (SDS) standard (IEEE STANDARD 1016). The document tries to address all the componentsneeded for the execution of theData analyticsworkload of the project and theway to report back theprogress to be reused in further executions or to dynamically react on real-time. Each component isdescribedintermsofitsexternalinterfacesanddependenciesonothercomponents.
Wewill alsodescribe the reasonsof the choicesdone in termsof technologiesand servicesused,andafine-grainworkplanforthenextsixmonths,leavingtherestoftheprojectinacoarsergraindefinitiontoletitevolvewiththetechnologychanges.
1.2 TargetAudienceThedocumentismainly intendedfor internaluse,althoughit ispubliclyreleased.ThemaintargetofthisdocumentistheglobalteamoftechnicalexpertsoftheEUBra-BIGSEA,includingWP3,WP4,WP5andWP6.ThisdocumentgoesbeyondtheQoSmonitoringtounderstandtheglobalarchitectureofWP3.ItdescribesthesoftwarearchitectureofWP3,themainbuildingblocks,therequirementsandthesoftwareavailable,as well as it proposes an architecture to instrument the development of policies, the deployment andexecutionofapplicationsandtheelasticitymeasures.
1.3 StructureTherestof thedocument isstructured into7mainparts.First,asummaryof therequirements fromtheusecasesispresentedanddiscussed.TherequirementsthataffectWP3areoutlinedandreferencedacrossthedocument.Then,section3describesthehigh-levelEUBra-BIGSEAinfrastructure,definingthescopeofeachoneofthemainlayersandtheirrelationtootherdocuments.Then,section4describesindetailtheQualityofService(QoS)InfrastructureasaService(IaaS),describingtheapplicationusecasesandthemaintechnologiesthatwillbeusedasbasisforthedevelopmentsofWP3cloudservices.Then,section5dealswith the monitoring architecture. Then, section 6 describes the policies that describe the resourceassignment. Finally, the document ends up with section 7, which describes the procedures for coding,sourcerepositoriesandcontinuousintegration,andsection8withtheconclusions.
2 DATAANDEXECUTIONREQUIREMENTSAdeeperanalysisoftherequirementsispresentedinD7.1End-UsersRequirementsElicitation.Thissectionincludes the requirements summary list for all the requirements that should be tackled from theWP3perspective.
Req# Description Priority WP
RE.1. UnrestrictedBatchjobs MUST WP3
RE.2. UnrestrictedBagofTasks MUST WP3
RE.3. QoSBatchjobs MUST WP3
RE.4. Deadline-basedscheduling SHOULD WP3
RE.5. Self-adaptingelasticity MUST WP3
RE.6. Short-jobs MUST WP3
Table1:ListofrequirementsrelatedtoWP3.
Fromthosehigh-levelrequirements,thisdocumentidentifies25technical-levelrequirements,describedinsection4.7, thatwilldrive the implementationof thesystem.Thosetechnical requirements refer toveryspecific details for five main categories: the management of resources, the scheduling of jobs, themanagementandcataloguingofcontainer images, theorchestrationanddeploymentofservicesandtheelasticity.Thesetechnicalrequirementswillbeintroducedinsections4.3to4.6.
- BigDataAnalytics services,which provide operators to process huge datasets andwhich can beintegrated in the programming models. Analytics services are characterized in the QoS cloudinfrastructure models of the underlying layer, which automatically (or explicitly driven by theanalyticsservices)willadjustresourcestotheexpectedworkloadandconsideringitsspecificities.
On top of those four blocks, applications are developed using the programming models and the dataanalytics extensions. Application developers are expected to use the programmingmodels andmay useotherfeaturesofunderlyinglayers,suchastheuser-levelQoSmetrics.
Figure 3 shows a more detailed schema of the architecture. More details will be provided in the nextsectionsandinthedeliverableD5.1.EUBra-BIGSEASoftwareArchitecture.
- Application binary and associated dependencies, embedded in a container. The applicationdependencies can be coded as a dependency file (“a la dockerfile”) or directly registering the
containerona repository.Thebasicexecutionunit is therefore thecontainer. Someapplicationsmaynotneedotherdependenciesdifferentfromtheonesfromtheprogrammingmodels,alreadyembeddedinthebasecontainerimages.
- QoSPolicies.Thedifferentapplicationswillbeanalysedintermsofdependencies,graphexecutionpath,resourcedemandandperformance.Whenanapplicationissubmitted,aninitialestimationofresourceswillbedefinedforeachexecutedcase.Reactiveelasticityoftheplatformwillensurethatenough resources are available for the application to start. The elasticitymodules can then usemonitoring data to dynamically update the resource allocation to ensure meeting the QoS. Forexample, the platform can enlarge the memory allocation for a Spark job if it is starving or aCOMPSsjobcanhavemorephysicalcomputingslotsifitisnotprogressingasexpected.
Mostof this information is inherent to theprogrammingmodel andpart of the specific algorithm tobeexecutedonit.Therefore,theuserwillnotneedtoprovideadditionalinformation,buttheanalysisoftherequirementsshouldderivethisinformationorthemethodstoobtainit.
- Programmingmodels.Theprojectwillstartfromtheexistingexpertiseofthedataanalyticsgroups,who already attain experience on popular execution environments. Application developers anddata scientists will express their requirements in the form of programming languages andframeworks.
- Execution patterns. Different use cases identified in this documentwill have different executionpatterns(event-based,bagoftaskbatch,interactive).
- Loggingandmonitoring.Theusecaseswillalsodefinemetricsthatwillbeloggedtofeedbackandadjust the static and real-time policies. Use cases should definewhich are themetrics they canexpose(additionallytothebasicresourcemonitoring,suchasCPU,memoryanddiskusage,ortheonesthatcouldbeobtainedfromthescheduler,asthejobwaitingtime)andwhicharerelevanttotheusecase.
Type Module,subprogram,datafile,controlprocedure,class,framework,service,etc.
License Licensemodel
Website URLtogetadditionalinformationorthecode.
Purpose Function and performance requirements implemented by the design component,includingderivedrequirementsthatrelatetotherequirementsoftheproject.
Function Whatthecomponentdoes,thetransformationprocess,thespecificprocessedinputs,the used algorithms, the produced outputs, where the data items are stored, andwhichdataitemsaremodified.
Dependencies Other components requiredby thecomponentsandhow this component isusedbyother components. Interactiondetails suchas timing, interactionconditions (suchasorderofexecutionanddatasharing),andresponsibilityforcreation,duplication,use,storage,andeliminationofcomponents.
Interfaces Detailed descriptions of all external and internal interfaces as well as of anymechanisms for communicating through messages, parameters, or common dataareas.
Data Internaldatarequiredbythecomponenttowork.
NeededImprovement
Description of the needed improvement to the tool, that are foreseen during theEUBra-BIGSEAprojectinordertofulfilluserrequirementsstatedinprevioussection.
4 QOSIAASThe execution requirements described in section 2 mainly address the execution of Batch jobs (RE.1), Bag of tasks (RE.2), QoS Batch jobs (RE.3), Short jobs (RE.6), Deadline-based jobs (RE.4) and the self adapting elasticity (RE.5). In order to address such requirements a job execution framework is needed. The Quality of services IaaS, main objective of WP3, focuses on the implementation of such a platform that could be capable of:
- Deploying and configuring the proper virtual infrastructure to run the data analytics jobs. This imply the specification of the software services, their dependencies, their configuration recipes and basic images.
- Orchestrating the execution of the workload on top of the virtual resources, leveraging the data locality and data parallelism.
- Registering the execution performance and logs to feedback and update further runs and to trigger elasticity rules.
- To implement elasticity at the level of the memory and the number of resource instances.
Those requirements should be addressed by a set of services, which may come from existing components in the literature and new developments from the consortium.
The section first describes the major workload cases and the lifecycle and the rest of the section describes the main components required for managing them.
4.1 ApplicationusecasesandLifecycleTheQoS IaaSaddresses twokindsofworkload:batch jobs thatare submitted through theprogrammingmodels(e.g.SparkorCOMPSs[R7])andinteractivejobsthatusetheDataAnalyticsconsole(e.g.OPHIDIA[R8]orSpark).
- Scenario1(modelfitting):Jobsarelaunchedwithapre-specifieddeadline.Jobsareexecutedoneby one and an ordered queue is maintained in a way that jobs can be executed within theirdeadlines.Whenanewjobissubmitted,thequeueisreorderedinawayalljobdeadlinescanbemetaccordingtothecurrentclusterconfiguration.Ifthisisnotthecaseortheexpectedmakespan(sum of execution time of all jobs) would be larger than a safety margin (e.g. the expectedexecution timewillbe10% longer than theestimatedone), thesystemcan increaseordecreasethe number of resources. The system periodically checks jobs execution by means of themonitoring system or specific calls to the monitoring API. If, considering data coming from themonitoring infrastructure thesystempredicts that thecurrent jobdeadlinecanbeviolated, thenclusterre-configurationistriggeredandresourcesareadded.Re-configurationisalsore-triggeredwheneveranewjobstarts.
- Scenario2(modelprojection):Wedealwithasustainedsubmissionrateofshortjobs(e.g.300jobsin5minutes,arrivinginanirregularpattern:from30to90jobsperminute).Eachjobwillrequire10seconds.Thesystemshouldquicklyallocateresourcesforthoseshortjobsinordertohavetheminimumoverhead (e.g. an overhead < 5 seconds per job).Wewant to have theminimum idletimeintheresources(e.g.<20%)toavoidwastingresources.
the information on the use of public transport from the previous day should be processed. If thisinformation isnotcomputedbythepeaktimeuserswillstartrequesting information,theresultsmaybeout-dated.Iftheanalysisisperformedquiteinadvance,theinformationmaynotbeup-to-dateenough.Sothejobshouldbeexecutedperiodicallyandendingclosetothepeakhours.
- DataanalyticbatchjobsthatcouldintrinsicallyexploitparallelismusingCOMPSs(Parallelregionsidentified by the COMPSs runtime are executed on multiple containers) or Spark jobs. Theseapplicationsmaybecharacterizedbytheexecutionpolicies,providingthemtheexpectedresourceprofileforitsexecution.
Along with the dependencies related to COMPSs, OPHIDIA or Spark, those applications may requireadditionalsoftwarelibrariesandcomponentscomingfromtheprogrammingenvironmentsused.
Therefore,theapplicationlifecycleinvolves:
- Defining theVirtual resourcedependenciesproperly.Thiswill require tocreatecontainer imagesthat can be either uploaded on public repositories or kept on the infrastructure resources. Apreferredmethodwill be to use dockerfiles,DockerHub and/orGitHubAutomatedbuild. Theseimages will be configured on top of the base images of the Spark, COMPSs and OPHIDIAenvironmentsandwillbelocallycreatedthroughDocker-composeifnecessary.
Thelifecycleoftheapplicationsgothroughdifferentstepsthatrangefromthedefinitionofthesoftwarearchitecture (software topology) that defines the application, the description of the dependencies, thedeployment on a resource pool, the configuration and contextualization of the instances, the actualexecutionoraccesstotheapplication,themonitoringoftheexecutionandthedynamicreconfigurationofthesystem.Figure4showsabasicdiagramofinteractionsforthislifecycle.
- The provisioning and orchestration of the resources. Section 4.3 includes a list of resourceprovisioningtechnologiesavailableandanalysestheselection.
- Theconfigurationandcontextualizationoftheapplication.Thisisdescribedinsection4.4.Itcoversthreemain alternatives: on-the-fly configuration of basic images, deployment of fully-configuredimagesandintermediatesolutions.
- Theexecutionofbatchjobs.Thisisdescribedinsection4.5.- Thedynamic reconfigurationof the resourcepool, described in section4.6, and addressingboth
horizontalandverticalelasticity.- Themonitoring and loggingof the resources and application services. Thiswas initially themain
4.2 OASISTOSCATheEUBra-BIGSEAplatformprovidesanentrypointtoitsfunctionalityviatheOrchestratorservice,whichwillfeatureaRESTfulAPIthatreceivesaTOSCA-compliantdescriptionoftheapplicationarchitecturetobedeployed. TOSCA (Topology and Orchestration Specification for Cloud Applications) [1] is an OASISspecification for the interoperable description of application and infrastructure cloud services, therelationships between parts of the services, and the operational behavior of these services. TOSCA hasbeen selected as the language for describing applications, due to the wide ranging adoption of thisstandardandtheavailabilityofsolutionsforbothOpenNebula[2]andOpenStack[3].
ThecoreTOSCAspecificationprovidesa languagetodescribeservicecomponentsandtheirrelationshipsusingatopologymodel,anditprovidesfordescribingthemanagementproceduresthatcreateormodifyservices using orchestration processes. The combination of topology and orchestration in a ServiceTemplate describes what is needed to be preserved across deployments in different environments toenable interoperable deployment of cloud services and their management throughout the completelifecyclewhentheapplicationsareportedoveralternativecloudenvironments.
A Topology Template consists of a set of nodes and relations. The nodes form a directed graph thatdescribe all the components of an application. Each node is represented by a node type and a set ofpropertiesthatdefinetheinterfacestomanipulatethecomponent.CustomToscatypesshouldbederivedfrom the normative types. Dependencies among Tosca types enable defining the dependencies andrequirementsinastructuredandportableway.ToscadeploymentplansinstantiatesuchToscatypeswiththespecificvaluesrequired.
A Tosca Topology template describe the interactions among the components (infrastructure,platform/middleware and application modules). Components are described in YAML [4]. Applicationarchitectscanmodelservices,policiesandrequirementsofanapplicationonaTOSCAtemplate,which isextendedwithadditional artifactsby thedevelopmentplans. TheTOSCA templatesareused to test anddeploytheapplication.
Figure5showsasampleTOSCAblueprint
tosca_definitions_version: tosca_simple_yaml_1_0 imports: - custom_types: <<URL to a yaml with generic custom_types.yaml>> description: > Sample TOSCA file topology_template: inputs: download_url: type: string default: <<URL to any specific data required for the installation>> node_templates: my_app: type: <<Existing generic type>> requirements: - host: my_app_prerequisites interfaces: Standard: configure: implementation: <<URL to yml file to install the software>> inputs: download_url: { get_input: download_url } my_app_prerequisites: type: <<existing generic type>>
4.3 ResourceManagementFrameworksA Resource Management Framework (RMF) is a multitenant component that manages distributedresources,allocatesthemtotherequestsperformedbyascheduler,executesaprocessinganddeallocatesthe resources once the execution finishes. This section describes the components required tomanage apoolofresourcestodeploytheapplicationcontainersthatsupporttheexecutionoftheapplications.Thosecomponentsdealwiththefollowingfeatures:
- Loadbalancing.Itshouldidentifythefreeresourceswherethecontainerswillbedeployed.- Fault tolerance. It should provide the ability to ignore faulty resources and to rearrange the
infrastructure.- Instance deployment and execution (Orchestration). Thismay be achieved through a scheduler
system,whichcouldbeindependentofthesystem.- Elasticity. It should enable enlarging or decreasing the resources allocated, despite that thiswill
- R01-TheRMFmustenableaschedulertoallocateresourcestoaspecifickernel.- R02-TheRMFmustsupportmultipletypesofworkloads,fromcontainerstoSparkjobs.- R03 - An application topology may involve several containers, which must be deployed in a
Purpose Anative clustering system forDocker. It turns apool ofDockerhosts into a single,virtualDockerhost.
Function Itenablescreatingapoolof resourceswheredockercontainerscanbe run.DockerSwarmprovidesDiscoveryServices,schedulingandbothaCLIandanAPI.
High LevelArchitecture
Eachhost runs a Swarmagent andonehost runs a Swarmmanager (on small testclusters this host may also run an agent). The manager is responsible for theorchestrationandschedulingofcontainersonthehosts.
Swarm canbe run in a high-availabilitymodeusingConsul or ZooKeeper tohandlefail-over toaback-upmanager (therecanbemultiplemanagers).Thereare severaldifferentmethodsforhowhostsarefoundandaddedtoacluster,whichisknownasdiscoveryinSwarm.Bydefault,tokenbaseddiscoveryisused,wheretheaddressesofhostsarekeptinaliststoredontheDockerHub.
ASwarmclusterrequiresanopenTCPportoneachnodeforcommunicationwiththeSwarm manager, to install Docker on each node and to create and manage TLScertificatestosecureyourcluster.
Type Amiddleware(setofdaemons/services)forclustermanagement
License Apache2.0
Website http://mesos.apache.org/
Purpose Mesos provides efficient resource isolation and sharing across distributedapplications (frameworks). Mesos can run on top of Virtual Machines and/or baremetaland/orDockercontainers.
Function Mesos requires computing resources to be assigned to it, so that it can deploydistributedapplicationsontheseresources.Mesosprovidesthefollowingmainfunctionalities:- Resource allocation/revocation/re-allocation: Mesos implements a pluggableresource allocation module architecture. By default, Mesos includes a strictpriorityresourceallocationmoduleandamodifiedfairsharingresourceallocationmodule. Advanced scheduling features like oversubscription are available sinceversion0.23.
- Performance isolation among framework executors running on the same slavenode through pluggable isolation modules. The default mechanism leveragescontainertechnologies.
- frameworkauthorizationimplementedthroughconfigurableACLsinJSONformatthatallow1)frameworksto(re-)registerwithauthorizedroles;2)frameworkstolaunch tasks/executors as authorized users; 3) Authorized users to shutdownframework(s)through“/shutdown”HTTPendpoint
- resourcereservation.Instaticreservation,resourcesarereservedforaparticularrole; Frameworks can use dynamic reservations (Mesos 0.24 on) to reserveoffered resources, allowing those resources to only be re-offered to the sameframework.Thisisespeciallyusefulifthetaskoftheframeworkstoredsomestateontheslave,andneedsaguaranteedsetofresourcesreserved,sothatitcanre-launchataskonthesameslavetorecoverthatstate.
- monitoring:Mesosmasterandslavenodes reporta setof statisticsandmetricsincluding details about available resources, used resources, registeredframeworks,activeslaves,andtaskstate.Thesemetricsareavailablequeryingthehttpendpointsexposedbythemasterandslavenodes.HeapstercanbeusedtomonitoraMesosinfrastructure.
- slave recovery: this feature allows1) Executors/tasks to keep runningwhen theslaveprocess isdownandallows2) a restarted slaveprocess to reconnectwithrunningexecutors/tasksontheslave.
TheMesosmasterisadaemonthatmanagesslavedaemonsrunningoneachclusternode. The high-availability and fault-tolerance of themaster can be achieved usingmultiplemastersandZookeeper.Theslaves registerwith themasterandoffer“resources” i.e. capacity tobeable toruntasks.Mesosusestheconceptofframeworkstoencapsulateprocessingengines.Frameworks creation is triggered by the schedulers registered in the system.Frameworks reserve resources for the execution of tasks. Mesos can reallocateresources to frameworks dynamically. Resource profiles are described as “roles”which define number of resources that can be allocated (something similar to the“flavours”). This ensures that there is no physical relation between resources andframework types (e.g. a Spark framework will only consume resources whensubscribedtoMesosanditcouldhavedifferentresourceseachtimeisexecuted).TheExecutorsrunontheMesosslavesandareresponsiblefor launchingtasks.Oneor more executors from the same framework may run concurrently on the samemachine. A dedicated class, “MesosExecutorDriver”, is used both to manage theExecutor’s lifecycle and to connect the Framework Executor to Mesos. There is alightweightexecutorforcontainerofbatchjobscalled“CommandExecutor”.Mesos provides the Scheduler interface to be implemented by each specificframework; this interface includesmethods to register, re-register, unregister withtheMesosmasterandtoacceptorrejectresourceoffers.The master decides how many resources must be offered to each frameworkscheduler according to a given organizational policy, such as fair sharing, or strictpriority. To support a diverse set of policies, the master employs a modulararchitecture that makes it easy to add new allocation modules via a pluginmechanism.Mesossupports internalmonitoringof frameworks,executorsandtasks,andMesosservicescanbemonitoredthroughasystemcalledSatellite.
Function YARNhas substitutedMapReduce for theprocessingof data inHadoop. It includesnowtwoseparatecomponents:TheResourceManager(oneperinstallation)andtheApplicationMaster(oneperapplication).
High LevelArchitecture
A YARN deployment has aResourceManager (RM) and aset of slave NodeManagers(NM) that constitute the jobenvironment. The RMmanages the resources andschedules the jobs. TheApplication Master (AM)negotiates the resources withthe RM and the NMs toexecute the tasks (a job inYARNcanbeaDAG). TheNMis the per-machine slave,which is responsible forlaunching the applications’containers, monitoring theirresourceusage(cpu,memory,disk,network)andreportingittotheRM.YARNhasaCapacityScheduler to runHadoopapplications ina shared,multitenantcluster, while maximizing the throughput and the utilization of the cluster. TheCapacityScheduler is designed to allow sharing a large cluster while giving eachorganizationcapacityguarantees.Thereisanaddedbenefitthatanorganizationcanaccessanyexcesscapacitynotbeingusedbyothers.Thisprovideselasticity for theorganizationsinacost-effectivemanner.
Function WithApacheMyriad,itispossibleto:- Runanoperationalapplications(includingthoserunninginDocker)side-by-sidewithananalyticapplications.
- AchieveHadoopmulti-tenancybyprovisioninglogicalHadoopclustersforeachuser.- YARN running as aMesos Framework,with resourcemanager and nodemanagersrunninginsideMesoscontainers.
- AbilitytolaunchmultipleYARNclustersonthesamesetofnodes.- Ability to deploy YARN Resource Manager using Marathon. This feature leveragesMarathon's dynamic scheduling, process supervision, and integration with servicediscovery(Mesos-DNS).
Mesos acts as the main resource manager.Myriad provides a Control Plane thatorchestratestheschedulingbetweenMesosandYARNschedulers.MesoswillspawnYARNNodeManagersasataskin theMesosNode (step1).TheNodeManagerregisters its capacity to the YARN ResourceManager (2), and then the Resource Managercanlaunchthecontainers(3).TheNodeManagercapacitycanbereadjustedtoenable other Mesos workload to profit unusedYARNresources.The control plane implements also horizontal scaling by getting information about thestarvationofresourcesfromYARNResourceManager.YARNclustersrunningonMesosthatcanallocateresourcesindifferentways:
- Static - Administrators can use an API or a GUI to add or remove nodemanagersorauxiliaryservicesliketheJobHistoryServer.
- Fine-grained - Administrators can provision thin node managers that aredynamicallyresizedbasedonapplicationdemand.
One of the workloads to be supported in EUBra-BIGSEA are batch jobs wrapped along with theirdependenciesoncontainers.Thebuilding,registrationanddownloadingofcontainersshouldbedoneinanefficientway, reducingcommunicationoverhead.This registry shouldbeavailable toall the resources inthesystemandupdateshouldbesimpleandautomatic.
PotentialUsage DockerHubcanbeusedtostorethebasicimagesofSpark,COMPSsandOPHIDIAandotherprocessingenvironmentsneeded.Thelocalregistryintheuser’sdeploymentwillstore the customised imageswith the additional software dependencies. Eventually,relevantandshareableimagescanbepulledintheDockerHubrepository.
Function Docker images are stored locally where theywill be run. A user can customize andstore amodified version of a container image locally, not sharing it with any otheruser.DockerHub is used toupload and share container images that areusedbymultipleusers and sites. Users “pull” the images that are available in this repository anddownloadthemontheylocalresources.Alternatively,ifthesourcecodeforaDockerimageisonGitHuborBitbucket,youcanusean“Automatedbuild”repository,whichisbuiltbytheDockerHubservices.
Inorder to facilitate themaintenanceof customcontainer imageswith theembeddeddependencies,anautomatedbuildprocesswillbeimplemented.Despitethecreationandmaintenanceofcontainerimagesisasimpleprocess,whensharedrepositoriesaremanagedthisprocesscouldleadtoincompletebuildsormanualsynchronization.Therefore,weproposetousetheDockerautomatedbuild.
Forthisprocess,theEUBra-BIGSEAGitHuborganization(https://github.com/organizations/EUBra-BIGSEA/)will be used in coordination with the EUBra-BIGSEA Docker Hub organization(https://hub.docker.com/u/eubrabigsea/). EUBra-BIGSEA developers will be invited to join bothorganizations.DockerHubenable creatingautomatedbuilds froma repository inGitHub.The repositoryshouldhaveadockerfileuploadedintheGitHubrepository.Anychangecommittotherepositorywillleadto the automatic creation of the docker image, which will be automatically registered in the properorganization repository of DockerHub. Users can pull the new images by typing “docker pulleubrabigsea/image_name”.
4.4 JobExecution(Scheduling)
The Resource Management Frameworks (RMFs) described in section 4.3 provide a way to share andallocate resources to multiple executions, even for heterogeneous traffic. On top of these systems,schedulersdispatchjobsontheallocatedresources.
Purpose In-memoryengine for large-scaledataprocessing.ApacheSpark isa fastandgeneral-purpose cluster computing system and an optimized engine that supports generalexecutiongraphs.
Function Developmentandexecutionof in-memorydataanalyticapplicationsbasedonanownprogrammingparadigm.
High LevelArchitecture
In a standalone cluster deployment, the clustermanager is a Sparkmaster instance.When using Mesos, the Mesos master replaces the Spark master as the clustermanager.
Spark can be used forbatch jobs throughspark-submit, whichcan use local, YARN orMesos resources,among others. Spark-submit can be used toexecute binariesremotely.
Spark-shellisaScalainteractiveconsolethatcanuseasback-endaMesoscluster.Thisway, one can execute data analytic operations and execute them interactively on aremotesystem.
Purpose Chronos is a suitable scheduler for periodic tasks that need to be triggeredperiodically or when other conditions are met. This is a sound alternative tonever-endingtasks,andfitsverywell thescenariosofperiodicdataextraction-transformation-load(ETL)jobs,Dataqualityanalysis,modelstuning,etc.
Function Chronos provides similar features (but more focused on periodic jobs) asMarathon.ItusesSSLauthentication,canbeusedtodefinegroupedjobs,enablejob retries and collects the historic performance of the executed jobs. It alsosupportslaunchingDockercontainers.
High LevelArchitecture
The high-level architecture is quitesimilartoonefromMarathon. Italsouses a JSON model to describe thejobs.
Jobs can be required to run onpredefined containers, and additionaldata canbe fetchedondemand. Jobsrun for a number of times (eveninfinitelyifrequired)andcanbescheduledtostartonaspecifictimeanddate.
Dependencies Chronos requires Apache Mesos 0.20.0+. Zookeeper is also required for highavailability.
Purpose It interacts with Mesos to submit batch jobs on a set of resources provisioned byMesos. It can submitbasic shell scripts, including theautomatic fetchingof files, andDocker-basedapplications.Marathon isespeciallywellsuitedfor longrunningservicejobs,thatneedhighavailability.
Function Jobs in Marathon are described in JSON documents that include the hardwarerequirements, the job execution code, URIs of dependencies and additionalinformation.Jobscanbeembeddedascontainers.JobsarecreatedandregisteredintheMarathonplatformfromtheirJSONdescriptionorusingtheWebGUI.Jobscanberestartedorstoppedasrequested.OtherinterestingfeaturessupportedbyMarathonare:
- SSLauthenticationoftheendpoints.- Applicationscalingondemand(bothincreasinganddecreasing).- Healthchecks.- Grouping of applications tomanage related applications that depend one on
Marathon is implemented into two components: theschedulerandtheexecutors.Theschedulercoordinatestheexecutionsandtheexecutorcontroltasks.
Mesos Master provides Marathon scheduler withresources (1). Then the executorwill launch a task (2)on the mesos slave provided by the master. Theexecutorwillupdatethestatusofthetask(3),fedintotheMarathonschedulerthroughtheMesosMaster(4).
4.5 ConfigurationandContextualization(andOrchestration)EUBra-BIGSEAwilldevelopacloud-basedplatformfordataanalytics.EUBra-BIGSEAwillproduceasetofservices to dealwith different types of applicationworkloads. One should be able to deploy only thosecomponents that are suitable for theapplication scenario. For example,one candeploya EUBra-BIGSEAinstrumented CMF with COMPSs as programming model for batch QoS jobs. Other configuration mayinvolveonlySparkjobs,includinginteractiveaccess.Therefore,thedeploymentofEUBra-BIGSEAshouldbeflexible,modular,platformagnosticandcapabletodealwithanunpredictedworkload.
Theconfigurationandcontextualizationprocessimplythesettingupofthevirtualresourcestobuilduptheapplicationenvironment.Itwill implytoinstallsoftwaredependencies,toconfigurethem,tosetupcrossinformationof IPs,directories,ports, etc. thatneed tobe sharedamong theapplicationcomponents, tocreate users, copy files, among other tasks. Contextualization deals with the core configuration of aninstance. Finally, orchestration relates to the coordinated deployment and reconfiguration of a set ofservicesthatformthetopologyofanapplication.
Applicationdependencieswillbeembedded in containers.Application-specific code is includedasbinaryexecutablefilescompiledwiththeCOMPSsruntimeorjarfilesfromSparkjobs.ASparkorCOMPSsjobwillexpect the Spark or COMPSs runtime dependencies automatically available in the container.Moreover,application-specificdependenciesmustbeindicatedsomehow.
Firstapproachwillproduceimmediatedeploymentofcontainers,butitwillrequireuserstobuilduptheirownimages.ThiscanbeautomatedthroughDockerfilesandautomatedbuildservices.Thiswillnotpreventfromadditional configuration steps tobe appliedon topof the container, for example, to configure thenetworkproperly.Secondapproachwillreducethemaintenanceoftheimages,whichcanbereducedtoaminimalsetandwill facilitatethedynamicreconfigurationofapplicationtopologies.Ontheotherside, itwillrequirealongerdeploymenttime.
In both cases, EUBra-BIGSEAwill needanorchestration system that could translate application topologydescriptionsinTOSCAintolocaldescriptionsthatcouldbetranslatedintoCloudManagementFramework-specificactions.
The InfrastructureManager (IM) isa tool thatdeployscomplexandcustomizedvirtual infrastructuresonIaaS Cloud by automating the Virtual Machine Image selection, deployment, configuration, softwareinstallationandmonitoringandupdateofvirtualinfrastructuresonmultipleCloudback-ends.
Thissectionelaboratesthecapabilitiesofthecloud-servicestoprovisionmoreorlessresources(instancesandmemory) according to the executionworkload. In the context of EUBra-BIGSEA, applicationswill beprovidedwiththeexpectedamountofresourcesneededtofulfillthejobexecutionconstraints.However,initialstaticallocationmaybecorrectedaccordingtotheexecutionprogress.
Thisadjustmentscouldhappenattwolevels:- A the level of Resource Management Framework (CMF), to adjust the infrastructure to the
workload.Multiplejobsmaycompetefortheresources,andeventually,thenumberofcomputingresourcesmaybe insufficient todealwith theworkload submitted.This information is knowna-priori, as each request will have the expected resources required. Moreover, in a fully-elasticenvironment,noresourcesmaybepreallocated,soinmostcases,theCMFwillhavetoreadjusttheinfrastructure.
- At the level of the Applications, to guarantee that the QoS is met. The monitoring system candetect a potential QoS break bymeans of the differentmetrics. Then, the reactive rules of themonitoringsystemshouldrequestadditionalresourcestobeallocatedtoanapplication.ThiswillrequiredealingwithboththeschedulerandtheCMF.
Bothverticalandhorizontalelasticityareconsidered:- HorizontalelasticitywillleadtoachangeinthenumberofresourcesintheCMFcluster.- Verticalelasticitywill leadtoachange in thememoryandCPUallocation limitsofanapplication
reducecosts.- Themonitoring systemdetects the need of increasing the resources of an application. It can be
triggered by a generic metric (CPU, network, memory, etc.) or an application-specific metric.Associatedruleswilltriggertheresourceincreasing.Resourcedecreasingislessimportantasitwillbeeitherdirectlytriggeredbythereactivesystemorunnecessary(e.g.cpuormemoryallocationincontainerizedapplicationsdoesnotimplypreemption).
Therequirementsofthesystemare:- R20-Automaticscalingupofresourceswhennewjobrequestsarise.- R21-Automaticdeallocationofresourceswhenidleforagivenperiod.- R22-Customizationpoliciesforwaitingtime,coolingtimeandsimultaneouspoweron,atleast.- R23 - Reallocation of resources for the execution kernels on the memory size and number of
Elasticity. Back-end component to manage the automatic power-on/off of virtualmachines.
Type Aserviceandaclientapplication.
License GPLv.3
Website www.grycap.upv.es/clues
Purpose CLUES is an energy management system for High Performance Computing (HPC)ClustersandCloudinfrastructures.
Function Themain functionof thesystem is topoweroff internalclusternodeswhentheyarenotbeingused,andconverselytopowerthemonwhentheyareneeded.CLUESsystemintegrateswith theclustermanagementmiddleware, suchasabatch-queuing systemoracloudinfrastructuremanagementsystem,bymeansofdifferentconnectors.
High LevelArchitecture
CLUESalso integrateswith thephysicalinfrastructure by means of differentplug-ins,sothatnodescanbepoweredon/offusing the techniqueswhichbestsuit each particular infrastructure (e.g.usingwake-on-LAN,IntelligentPlatformManagement Interface (IPMI)orPowerDeviceUnits,PDU).
Although there exist some batch-queuing systems that provide energysaving mechanisms, some of the mostpopular choices, such as Torque/PBS,lack this possibility. As far as cloudinfrastructuremanagementmiddlewareis concerned, none of the most usualoptions for scientific environmentsprovide similar features. Theadditional advantageoftheapproachtakenbyCLUESisthatitcanbeintegratedwithvirtuallyanyresourcemanager,whetherornotthemanagerprovidesenergysavingfeatures.
Currently,CLUEScountswithconnectorsforintegrationwithsomeofthemostpopularbatch-queuingsystems(suchasTorque/PBSorSunGridEngine)andwithOpenNebulaand OpenStack which are two of the best known cloud infrastructure managementsystemswithinthescientificcommunity.
Function Itconsistsofasetofrecipesandacommand-lineinterface(CLI)usedasaclientfortheIMinordertodeployacustomizedfront-endnodeofavirtualclusterthatfeatures:i)aninstanceofanIMtoprovisionforadditionalcomputingresources(workingnodes);ii)CLUES,implementingtheelasticityrulesconsideringthestateoftheLocalResourceManagement System (LRMS) and iii) the specific configuration for the virtual clusterrequiredfortheexecutionoftheapplicationsthatwillberunonthecluster.EC3isalsoofferedasafreeonlineservicetodeployon-demandelasticvirtualclustersonAmazonWebServices,OpenNebulaandOpenStack.
Inthefigure,theCLIofEC3isusedtocontacttheIMinordertodeployandconfigurethefront-endnodeofavirtualclusteronaCloudinfrastructure.Thefront-endnodeisconfiguredwithanotherinstanceoftheIMtogetherwithCLUES,whichisinchargeofimplementingtheelasticityrulesbyinterceptingthejobssubmittedtotheLRMS(e.g.SLURM, PBS/Torque, etc.) and deciding when additional worker nodes should bedeployeddependingon theamountandcharacteristicsof the jobsqueuedupat theLRMS. Optional components in the architecture of EC3 that could be of interest for
EUBRa-BIGSEA are the checkpointmanager (CKPTManager) andBLCR (Berkeley LabCheckpoint Restart), which enable to introduce efficient and cost-effective usage ofspotinstances,availableforAmazonWebServices,tocheckpointthestateofthejobsandautomaticallyrestartthemonapoolofworkernodesdeployedonspotinstances,whichcanbeterminatedanytime.Users connect via SSH to the front-end node and submit jobs as usual. The virtualcluster will deploy additional worker nodes as required, and integrate them on theLRMS without user intervention, in order to cope with increased workload of jobs.Workernodeswillbeterminatedwhentheyarenolongerrequired.
Data EC3 relies on the templates/recipes, written in IM’s native language, called RADL(Resource&ApplicationDescriptionLanguage)whicharecurrentlyavailableatGitHub.ItproducesasoutputtheIPofthefront-endnodethatcanbeaccessedviaSSH.
Purpose It enablesoversubscription inmemory for theVirtualMachines runningonaCMFandimplementsmemoryballooningandautomaticlivemigrationofVMs.
Function CloudVAMPimplementsthreemainfunctions:- Dynamic Memory Resize for VMs. Cloud users tend to overestimate the
memory requirements of their applications/services and templates typicallyrepresent upper bounds for application requirements, thus wasting memoryallocated to the VM. CloudVAMP monitors actual memory usage anddynamically resizes the memory allocated to VMs, relying on the memoryballooningsupportprovidedbytheKVMhypervisor.
- EnableOversubscription.ThestolenmemoryfromtheVMs isexposedasfreememory to theCloudManagementPlatform (CMP).TheCMP's scheduler candecidetoallocateadditionalVMsonthephysicalhosts..
- PreventMemory Overload by LiveMigration. Oversubscription may result inmemory overload of the physical host if no preventive measures areconsidered. For that, CloudVAMP prevents memory overload via the livemigration of VMs across the physical nodes, as supported both by KVM andOpenNebula. No downtime is introduced for VMs and memory overload isprevented,thusmaintainingtheLevelofService.
High LevelArchitecture
ThearchitectureofCloudVAMPconsistsofthreecomponents:- The Cloud Vertical Elasticity Manager (CVEM). An agent that analyzes the
- TheMemory Reporter (MR). An agent that runs in theVMs and reports to amonitoringsystemthefree,usedmemoryandusageoftheswapspace,bytheapplicationsintheVM.ThisinformationmustbeavailableforCVEM.
ThepreviousfiguredepictsthearchitectureofthesystembasedonaOpenNebula(ONE) implementation. OpenNebula requires a cluster-based installation in whichthemainservicesareinstalledinthefront-endnodewhereastheVMsaredeployedontheinternalworkingnodes,wheretheKVMhypervisorhastobeinstalled.
Theabovesectionshaveanalysedasetofexistingtechnologiesandsomeimprovementsontopofthem.Some of the components complement, meanwhile other components compete. This section tries toanalyseprosandconsoftheabovecomponentstobuildupthecloudexecutionservicesoftheQoSIaaS.
- Interactive jobs . Mesos will allocate the resources required to submit the applicationDAG. Mesos scheduler API [http://mesos.apache.org/documentation/latest/scheduler-http-api/] could be used to allocate resources and Mesos executor API[http://mesos.apache.org/documentation/latest/executor-http-api/] to access theresources.
- Directlymodify the resourceallocationofCPUandmemory toanactive taskbyupdating theconfiguration in Chronos, Marathon and Spark, by means of a direct interaction from thenotificationsofthemonitoringsystemandactingontheMesosCMF.
- Scale up the number of instances in Marathon, by means of a direct interaction from thenotificationsofthemonitoringsystem.
- Use CLOUDVAMP under the scenes to dealwith themanagement of VMs at the level of theCloudManagementFramework.Thiswillbetransparenttotheapplicationdevelopers,butitwillexposemorememorythanavailable.
Inorder to interactwith the single submissionend-point,wewilldefineamacro-APIandacommon jobdescriptionthatcouldbemappedtoMarathon,Chronos,Sparkorsimilarschedulers.TheAPIwillbeaRESTAPIthatwillinteractwiththetwomainentities:theResourceManagementFrameworkandtheScheduler.The rest of the serviceswill use this API to submit jobs, retrieve their status, kill jobs, get the status ofresourcesandreallocatethem.
Figure6showsaJSONstructureproposaltodefinejobs.Thesamestructurecouldbeusedtoretrievetheinformation-extendedwithadditionalfieldssuchastheactualmemoryusage,theactualCPUusage,theapplication id and the values of the parameters we have left the scheduler to choose, such as portallocation.Itcanbeusedtoupdateconfigurationswhenreasonable.{ "type": "CMD", "name": "my_job_name", "deadline": "2016-06-10T17:22:00Z+2", "periodic": "R24P60M", "expectedduration": "10M" "container" : [ "type": "DOCKER", "image": "eubrabigsea/ubuntu", "forcePullImage": true "volumes": [ {
YEAR-MONTH-DAY,HOUR:MINUTE:SECONDTimezone(ZisUTC).Optional.- "periodic". It indicates if the job has to be repeated or not. Rn indicates ‘n’ repetitions and ‘R’
- "expectedduration",describestheexpecteddurationofajob.Optional.- "container", describes the configuration features for a container-based job.Mandatory for non-
Spark jobs. The fields are: "type”, describes the driver; "image", indicates the container image";"volumes": indicate thevolumemappingof thecontainer;“portMappings”, list theportmappingbetween the container and the host. No values indicate either no mappings or the automaticselectionofports.
- "environmentVariables",definespairsofenvironmentvariablenameandvalue.Optional.- "cpu", defines the allocation of CPUs (could be a float number) estimated for the job. The
framework should allocate enough resourcesor queue the job. Similarmeaning for thememoryanddisk.Units inKBytes(K),MBytes(M)andGBytes(G).This informationshouldbeprovidedbytheproactivepoliciessystem.Mandatory.
5.1 CloudcomputingmonitoringMonitoring in cloud environment differs over traditionalmodel,where you have a dynamicworkload incontrastofastaticsetofassets.Theoldtraditional infrastructureexpectedthatnewserverswererarelyinstalled and old ones decommissioned sometimes. The whole infrastructure was meant to be"permanent".Cloudcomputingincontrastisadynamicandfluidenvironment.Serversarecreatedanddestroyedallthetime, adjusting the increased traffic on the service it runs or reducing the amount of resources to savemoney.A cloud monitoring system should reflect these dynamic aspect while providing essentials informationswhich helps providers to properly plan their infrastructure capacity and resourcesmanagement; ensureSLAsandoverallQoS,offeredthroughandbythecloud;identifybottlenecks,failuresorproblems;andbillresourcesusage.
5.2 Requirementsformonitoringservice
Accordingto[R09],manymonitoringsystems(MS)havenotaddressedtherapidlychanginganddynamicinfrastructure seen in service clouds. Authors then have determined main requirements for monitoringcloudscenarios:
Elasticity - in order to handle drastic changes ofmonitored environments,MS should support upsizinganddownsizingofmonitoredresources(e.gagents,probes,etc.).
Migration - virtual resources may be moved from one physical host to another. All previous dataassociatedtothesevirtualresourcesmustbeindependentofthephysicalhost.
Adaptability - it should be minimally invasive and adapt itself to various computational and networkloads.
Timeliness - events andmeasurementsdata shouldbeavailableon time for their intendeduse [10].Aconsumer may require updated data from a producer in order to execute an action but theinterdependencyofTimelinesswithotherpropertiesof themonitoringsystem,suchasElasticity,AutonomicityandAdaptability,implieschallengesortrade-offsbetweenopposingrequirements.
Comprehensiveness - it should supports different levels of abstraction. Such levels includemonitoringphysicalandvirtualresourcesandserviceapplications.Ineachlevel,differentmetrics,kindofdataand probes are available. It is desirable to support isolation of environments, by segmentedauthentication,authorizationandresources,asamulti-tenantsystem.
High availability - it should employ a high availability strategy, such as data replication/redundancy,workloadbalancingandsharding.
Accuracy -measuresprovidedbymonitoringsystemareaccuratewhentheyareascloseaspossibletothe real value measured and they satisfy what the application considers accurate (applicationdependent).
Extensibility - newmetrics and probesmay be added to themonitoring system, including those onesdefinedintheapplicationlevel.
Alerttriggering-ifapre-configuredconditionissatisfiedinmonitoringsystem,actionsmaybeexecutedin the monitored resource or system wide. Actions include execution of programs, sendingmessagestooperators,dynamicallyadjustmeasurement,etc.
- Monitoring agent - runs on monitored resource, providing probes results to monitoring system;perform actions coordinated by monitoring system and triggered by some configured criteria.Measurementsincludessystemmetrics(e.gCPU,disk,memoryusage,etc.),Nagiosplugins,statsdandmanychecksforservices(e.gDBMS,WebServers,customapplications,etc.).
- Monitoring API: An API, preferably RESTful, focused on get, set and querying metrics, providingstatistics aboutmetrics, alarmmanagementandnotifications. ThisAPIwill beusedby someEUBra-BIGSEAcomponents,speciallythosedealingwithWP3dynamicandstaticresourceallocation.
- Messagequeue:A component that receivespublishedmetrics frommonitoringAPIandalarmstatemessages from threshold engine. Message queue decouples system components and defines acommoncommunicationbusbetweendifferentsystemcomponents.
- Persister services: Consumes metrics and alarms from message queue and store them in its owndatabase.
- Metricsandalarmsdatabase:Acomponent, ingeneral,aDBMS,whichprimarilystoresmetricsandalarmshistory.SomeNoSQLsolutionsprovidedifferentparadigms inrespectofhowdata is logicallyorganized.Forinstance,CassandraNoSQLdatabaseallowsmodelingdatainacolumn-orientedformat,organised in a such way that it is very efficient to retrieve data, specially those represented astemporalseries.Forunstructureddata,suchaslog(syslog,applog),ElasticSearchisusedtoindextext.
- Notification engine: Consumes alarm state transitionmessages from themessagequeue and sendsnotifications,suchasemailsorremoteexternalAPIevocationforalarms.
- Analytics engine: Consumes alarm state transitions andmetrics from themessage queue and doesanomalydetectionandalarmclustering/correlation(optional).
- Configuration database: A component that stores the configuration and other information in thesystem.
- Dashboard monitoring interface: An application (preferably web-based) that includes configureddashboardsandreports. Itallowsusersaccessmetricsvalues,alarmstatusandresourcesallocation,showingdataascharts,tablesandanyotherkindofvisualisationapplicable.
Monitored resource may be running in different hardware abstractions: bare metal, virtual machine oroperationalsystemcontainerand,here,hostreferstoanykindoftheseabstractions.Independentlyoftheabove,monitoringsystemshouldbeabletointeractwithitsagents.Therewillbeaminimumsetofmetricsandtheirvariationsorderivationssupportedbymonitoringsystem:
- CPU - Discover if a host’s CPU is being heavily utilized by the kernel, app running code, or otherprocessesonthesamehost.Itcanbesegmentedinsystemandusertime.
- Memory-Knowwhenanapplicationisconsumingtoomuchmemory,andhowmuchof it isbeingconsumedbytheoperatingsystem.Virtualmemoryisalsoconsidered.
- File system -Monitor file systems, detectingout of spaceor available i-nodes, bytesor anyotherdimension. Also may detect unavailability or mounting state changes (for instance, read-onlymountingafterfailure).
- Disk - Measure input/output operations per second (IOPS) and throughput, in order to identifybottleneckswithgreaterprecision.
- Network - Detect network errors, collisions, overruns, and dropped packets, in order to diagnosenetworkinterfaceproblems.Maybeusedtodetectofflineresources(pingliketests).
- Uptime-Howlongresourcehasbeenrunning.
5.3.2 Supportedapplicationmetricsandlogging
Each application should be able to define its own metrics, related to security, performance, capacity,uptime,throughput,servicelevelagreements(SLAs),usermetricsoranyotherinternalvalue.Byusingthemonitoring system proposed API, applications can create, remove, update and query metrics,measurementsandalerts.MetricsandtimeseriesexclusivelyusedbytheapplicationitselforunrelatedtoWP3mustnotbestoredinmonitoringsysteminfrastructure.
- Pingcheck.Monitorsservercommunicationandresponsetimes.- HTTPcheck.Getspecificcontenttoverifythatuserscanseeandinteractwithapplications.- HTTPScheck.MonitorsthevalidityoftheSecureSocketLayer(SSL)certificate- SSH check. Confirms that the target server’s secure shell (SSH) protocol is running and accepting
Monitoring systemwill support basicmanagement of application logs. Applications can submit logs to acentralmonitoringsystemcomponent, responsible forprocessing,extractingand indexingof logrecords.Eachlogrecordmusthaveasetoffieldsthatincludesdate/hour,severity,nameoftheapplicationandthemessageofthelog.
5.4 TechnologyevaluationThis section analyses solutions for the monitoring of cloud infrastructures.We first detail tools for thecollection,storageandretrievalofmonitoringmetricsand,then,discussapproachesformanaginglogs.
Function Zabbixisanagent-basedmonitoringsystemthatcancollect,storeandmanageawiderangeofsystemmetricsrelatedtohosts,Virtualmachines,applicationsandservices.Zabbix consumes more resources than other environments, but provides a morecompleteUIandasimplerwaytocodeprobesandmonitoringKPIs.
Data Configuration,metricsandtriggerhistoryarestoredinaMySQL.
NeededImprovement
Zabbixhasauto-registrationfeaturesincev2.0.Zabbixagentscanregisterthemselveswithout manual intervention. Although, auto-registration is not well documented.ResourcesremovalisnotautomaticanditwouldneedinteractingwiththeAPI.
5.4.2 InfluxDB
InfluxData is thealerting, visualizationandbackend forbuildingcustommonitoring solutions for servers,sensors,storageappliances,networkinfrastructure,apps,logsandmore.
Sensu is an open-source monitoring framework that allows organizations to compose comprehensivemonitoring & telemetry solutions thatmeet their unique requirements. Provides a platform focuses onwhattomonitorandmeasure,ratherthanhow.
Identification SensuCore
PotentialUsage Monitoringandtelemetrysystem
Type Infrastructureandapplicationmonitoringandtelemetrysolution
License MITLicense
Website http://sensuapp.org/
Purpose Controlandintegratemonitoringresources
Function Integrates monitoring infrastructure, service & application health, and business
KPIs.ItsupportsdeploymentviaPuppet,ChefandAnsible.Extensibleframework(includingamessagebus,eventprocessor,monitoringagent,anddocumentedAPIs),andmanyavailableplugins.Integrateswiththetoolsandservices(e.gemails,PagerDutyalerts,Slack,HipChat,IRCnotifications,etc.).Service checks provide status and telemetry data, and event handlers processresults. Hundreds of plugins are available for monitoring the tools and servicesalready inuse.Pluginshaveavery simple specification,andcanbewritten inanyprogramminglanguage.
High LevelArchitecture
Clients executes servicechecks, plugins/scriptsfeeding broker(RabbitMQ) with data,which will be consumedby server/API andprocessed by handlers,mutators,etc.,performingspecific actions, such assending data to a timeseriesdatabaseorsendinganalerte-mail.Events and check statusare stored and query onRedis.
Dependencies Redis,RabbitMQ,Ruby(embedded)
Interfaces RESTfulHTTPAPI;SimpleUXdashboard
NeededImprovement
Dashboard interface is quite simple and lack features like, auth, roles, customsettings.
5.4.4 OpenStackMonasca
Identification OpenStackMonasca
PotentialUsage Completemonitoringsystem
Type Monitoring-as-a-servicesolution
License ApacheLicense,Version2.0
Website https://wiki.openstack.org/wiki/Monasca
Purpose Controlandintegratemonitoringresources
Function Fullsolutionformonitoringinfrastructure,service&applicationhealth,andbusinessKPIs.IntegrateswithOpenStackcomponentsfororchestrationandaccesscontrolandscoping.
Function Multi-threadingTCP, SSL, TLS, RELPMySQL, PostgreSQL, Oracle and moreFilter any part of syslog messageFully configurable output formatContent-basedfiltering
High LevelArchitecture
Rsyslog can act in various modes. It ispossible towrite logsmessages into thelocal hard drive and/or forward logsmessages over network to a central logserver. On central log server, messagescouldbe filteredbycontentandwritteninto different local harddrives/databases.
LogmessagesaresendviaTCPorUDPtocentral server which will process andcould do things like splitting intobranches.
Function ElasticSearch- Distributed,scalable,andhighlyavailable - Real-timesearchandanalyticscapabilities - Document-Oriented & Full text search functionality, with powerful query
5.5 SolutionevaluationThis section analyses the different complete architectures and components for the monitoring andproposesasolutiontobeusedinEUBra-BIGSEA.
Configuration is done through JSON files, defined in sensu-server and sensu-client. Checks results andmeasurements are collected via scripts running onmonitored resources. All data are sent to amessagequeueandconsumedbysensu-server,thatwillanalyze,exhibitinformationandperformsomeaction(sendemail,proxyinformation,etc).
IndependentfromSensu/InfluxDBthelogpartisexecutedbyacombinationbetweenrsyslog,ElasticSearch,Logstash and Kibana. Rsyslog will sent log application to Logstash server, that will parse and extractmeaningful informationfromlogtextandstore intoElasticSearchforfurtheranalyses.KibanawillexhibitthevariouskindofanalysesthatwecandonewithElasticSearch.
Monasca is a fullmonitoring solution that bind a few softwares tomake itwork. It uses a RESTAPI forinteracting and receiving checks results andmeasurementsdata.All data areput inmessagequeueandconsumed by engines (alarm engine, notification engine, etc.). Processed data is stored in database forfurtheranalyses,visualizationandactionsexecution.MonascaisdevelopedinPythonandJavawhichusesApache Kafka as message queue, Apache Storm for real time data computation and Zookeeper toorchestratecomponents.
Proposed API is RESTful JSON and may be used by implementing clients in different programminglanguages. Clients will use HTTP or HTTPS protocols and all requests must be authenticated by using aspecialHTTPheader.CompletedocumentationoftheAPI,includingalltechnicaldetails,isoutofscopeofthisdocument.
/v1/metrics/measurements Allow to store measurements values with a timestamp for a specificmetricorasetofmetricsandtoqueryoriginaloraggregatedvaluesformeasurements.
/v1/groups Allow adding monitored resources as group members. Add metricsand/oreventstogroupmembers
5.7 ConclusionSensu and Zabbix solution are mature and working solutions, but choosing Monasca offers a greatopportunitytocontributetoaworld-wideprojectoncloudplatformsandestablishacloudstandardfortheEUBra-BIGSEAproject,asitwillbecomeapartofOpenStack.TheOpenStackecosystemisrichofmemberfromdiverseareas,academyand industry,whichcouldprovideusgoodpracticesandstateof theart inmonitoringsolutions.TheshorttermroadmapoftheMonascaprojectalsoalignswellwithEUBra-BIGSEAas itconsiderstheusageoftheELKframeworktomanagelogs.Monascaalsoprovidesstandardpatternsforcreatingmetricsandextendthewholemonitoringsystem.Support forhigher levelsofabstractions isdonethroughtheusageofStatsdorbydirectlypostingtoitsmetricsAPIs.MonascaWebhooknotificationscanalsobeusedtotriggeractionsinthereactiveelasticiymodules.
Monasca integrates with Openstack Keystone, a federated authentication and authorization solution,offeringthepossibilityoflimitingthevisibilityofthemetricsbyscopingthemtoaspecificproject.Monascaalso provides all the essentials requirements as measurements collection, high availability, extensibility,alarmstriggeringandarichinterfaceforcreating,configuringandvisualizealarms.
6 TECHNICALPROCEDURESThe project will not impose themigration of the components which already are available on a specificrepository to a central one. However, the use of a common repository for the new developments isencouraged.
- An EUBra-BIGSEA Docker Hub organisation can be created to store the corresponding basiccontainer images, and any other image derived from the application analysis(https://hub.docker.com/r/eubrabigsea/).
7 CONCLUSIONSThis document has gone through a thorough review of the components that will form the QoS cloudservicesarchitecturefortheBigDataanalyticsplatformdevelopedinEUBra-BIGSEA.Themainfocusofthedeliverable was initially to focus on themonitoring architecture, but it has been extended to draw thewholelayerofservicesfortheexecutionandmonitoringofcloudservices.
The document proposes the use ofMesos as the basic foundations for themanagement of distributedresources,asitisawidelyusedcomponenttoprovideisolation,toavoidfragmentationofdatacentresandto achieve high-availability and reliability. Mesos will be enhanced with the capability of automaticallyincreasingresourcestoprovidearealadaptationofworkloadtopoweronresources,whilebeingagnosticto the upper layers. The project identifies three types of workloads, persistent, periodic batch andinteractive jobs,whichwillbeservedbydifferentschedulers.Persistent jobswillbeservedbyMarathon,periodicjobsbymeansofChronos,andinteractivejobsthroughinteractiveshells(e.g.sparkshells).Thoseschedulerswill deploy frameworks thatwill embed the executable services and negotiate the resourceswithMesos.ThoserequestswillbeinterceptedtoguaranteetheavailabilityofresourcesinMesos(e.g.byaddingmoreresourcestothesystem).
QoS cloud services will deal with the reactive elasticity of the Mesos cluster and the physical (virtual)resources that Mesos base on. Frameworks could request additional resources in case of applicationstarvationanddependingontheprofileof theapplication.Then, themonitoringsystemwill triggersuchactions.
EUBra-BIGSEAwillcontributetothisecosysteminseveralaspects:First,itwillprovidereactiveelasticitytotheresourcemanagementplatform,whichwilluseTOSCAasstandardspecificationtoenhanceportability.Thiswill enableMesos clusters to grown and shrinkwithin datacentre boundaries. Next, itwill create asinglesubmissionpointthatwilldecideuponthedifferentschedulersbasedonthe jobfeatures.Third, itwillprovidereactivepoliciesconnectedtothemonitoringsystemtoensurethatthedeadlinesaremet.Andfourth, itwill integratemonitoringandnotificationsystemswiththeproactivepoliciesto learnfrompastexecutions,reducingtheneedoffine-tuningofthereactivepolicies.
[R9] Clayman, Stuart et al. "Monitoring Service Clouds in the Future Internet."Future Internet AssemblyApr.2010:115-126.
[R10]Wang, Chengwei et al. "A flexible architecture integrating monitoring and analytics for managinglarge-scale data centers." Proceedings of the 8th ACM international conference on Autonomiccomputing14Jun.2011:141-150.
