Containers• Lightweightandareproduciblewaytorunapplica8ononheterogeneousnodes.
• Separatestheapplica8onfromthenodeOS.
• PopularContainerTechnologies• Docker
• Popularintheenterpriseworld.• Bydefault,applica8onlaunchedincontainerrunasroot-Aconcernwhenrunningonsharedinfrastructure
• Singularity• PopularinHPCenvironments.• Isruninuserspace.
Why use Containers for your workflow?Tradi8onalwayofreferringuserexecutableinPegasus• Jobsintheinputabstractworkflow(DAX)refertologicaltransforma8ons.• Usersdefinemappingoflogicaltransforma8ontoactualexecutableinaTransforma8onCatalog
Executablestagingworksifexecutableissta8callylinked,OR
iflibrariesareinstalledonthenodesfordynamicallylinkedexecutables
executables descriptionlistofexecutablesloca8onspersite
physical executables
... # This is the transformation catalog. It lists information about each of the # executables that are used by the workflow. tr ls { site compute-site{ pfn "/bin/ls” type "INSTALLED” arch "x86_64" os "linux" osrelease ”centos" osversion “7" } } ...
transformation typewhetheritisinstalledor
availabletostage
mappedfromlogicaltransforma8ons
Why use Containers for your workflow?Tradi8onalwayofreferringuserexecutableinPegasus• PegasusmatchestheaRributesofanexecutabledefinedinTransforma8onCatalogagainsttheaRributesspecifiedforsiteinSiteCatalog.
• Thisapproachworksfineifyoursiteismadeofhomogenousnodes• However,problemsoccurwhen
① yourunonasitewithhetrogeneousnodesandyourjoblandsonanodewhereOSisincompa8blewithyourexecutable
② Applica8onisamis-matchtothecomputenodeenvironment
• Installlibrariesinyoursharedspaceandmakesureenvironmentreferstothoselibraries• Needcoopera8onfromSiteAdmins.OnOSG,youcaninstallthingsinCVMFS• TensorFlowrequiresspecificpythonlibrariesandversions.SomelibrariesmaybeeasytoinstallonlatestUbuntu,butnotonEL7
Pegasus Container Support• IntroducedinPegasusRelease4.8
• SupportforbothDockerandSingularity
• UserscannowrefertocontainersintheTransforma8onCatalogwiththeirexecutablespreinstalled.
• Userscanrefertoacontainertheywanttouse.However,theycanletPegasusstagetheirexecutabletothenode.
• Usefulifyouwanttouseasiterecommended/standardcontainerimage.• Usersareusinggenericimagewithexecutablestaging.
Specifying Containers in Transformation Catalog
containerReferencetothecontainertouse.
Mul8pletransforma8oncanrefertosamecontainer
image
... tr pegasus::keg{ site isi { pfn "/usr/bin/pegasus-keg arch "x86" os "linux" osrelease ”centos" osversion “7" # INSTALLED means pfn refers to path in the container. # STAGEABLE means the executable can be staged into the container type "INSTALLED" #optional attribute to specify the container to use container "centos-pegasus" } } cont centos-pegasus{ type "docker” image "docker:///centos:7" # optional site attribute to tell pegasus which site tar file # exists. useful for handling file URL's correctly image_site "optional site" # environment to be set when the job is run in the container # only env profiles are supported profile env "JAVA_HOME" "/opt/java/1.6" } ...
URLtoimageinadocker|singularityhubOR
toanexis8ngdockerimageexportedasatarfileorsingularityimage
typeCanbeeitherdockerorsingularity
Data Management for Containers
• Userscanrefertocontainerimagesas§ DockerorSingularityHubURL’s§ DockerImageexportedasaTARfileandavailableataserver,justlikeanyotherinputdataset.
• Ifanimageisspecifiedtoberesidinginahub
§ Theimageispulleddownasatarfileaspartofdatastage-injobsintheworkflow§ Theexportedtarfileisthenshippedwiththeworkflowandmadeavailabletothejobs§ Mo8va8on:AvoidhicngDocker/SingularityHubrepeatedlyforlargeworkflows
• Pegasusworkerpackageisnotrequiredtobepre-installedinthecontainer§ Ifamatchingworkerpackageisnotinstalled,therequiredworkerpackageisinstalledatrun8mewhencontainerstarts
Container Execution ModelContainerizedjobsarelaunchedviaPegasusLite
• Containerimageisputinthejobdirectoryalongwithinputdata.• Loadsthecontainerifrequiredonthenode(applicableforDocker)• RunascriptinthecontainerthatsetsupPegasusinthecontainerandlaunchesuserapplica8on
• Shutdownthecontainer(applicableforDocker)• Shipouttheoutputdatageneratedbytheapplica8on• Cleanupthejobdirectory
• Tradi8onalshared-fsapproachdoesnotsupportcontainers.
Directories Mounted• OnlythejobdirectorywherePegasusLiteplacestheinputsismountedinthecontainer
• Docker–Mountedas/scratch• Singularity–Mountedas/srv
• PegasusLiteensuresthatuserapplica8onislaunchedinthedirectorymounted
• ConsistentwiththePegasusmodelofensuringthatuserjobislaunchedindirectorywhereit’sinputdataexists.
User Running in the Container• Singularitycontainersalwaysruninuserspace.• Docker
• Pegasusbeforelaunchingtheuserapplica8on• CreatestheuserinthecontainerthatisthesameastheuserunderwhichthejobislaunchedbytheLocalResourceManagerontheremotenode
• Whydowedothis?
• Bydefault,Dockerrunsuserapplica8onasroot• NotrecommendedforHPCenvironment
• Createsproblemswithstagingtheoutputscreatedinthecontainer
Reference• Documenta8on
• hRps://pegasus.isi.edu/documenta8on/containers.php
• Example
• hRps://github.com/pegasus-isi/montage-workflow-v2/• Scriptexample-dss-containers.shwillrunthemontageworkflowjobsinacontainerpulledfromthesingularityhub
Pegasus Automate,recover,anddebugscien8ficcomputa8ons.
Thank You
Questions? [email protected]
KaranVahi
RafaelFerreiradaSilva
RajivMayani
MatsRynge
EwaDeelman
Meet our team
est. 2001