Container Support in Pegasus 4.8.x

KaranVahivahi@isi.edu

h"ps://pegasus.isi.edu

Containers•  Lightweightandareproduciblewaytorunapplica8ononheterogeneousnodes.

•  Separatestheapplica8onfromthenodeOS.

• PopularContainerTechnologies•  Docker

•  Popularintheenterpriseworld.•  Bydefault,applica8onlaunchedincontainerrunasroot-Aconcernwhenrunningonsharedinfrastructure

•  Singularity•  PopularinHPCenvironments.•  Isruninuserspace.

Why use Containers for your workflow?Tradi8onalwayofreferringuserexecutableinPegasus•  Jobsintheinputabstractworkflow(DAX)refertologicaltransforma8ons.•  Usersdefinemappingoflogicaltransforma8ontoactualexecutableinaTransforma8onCatalog

Executablestagingworksifexecutableissta8callylinked,OR

iflibrariesareinstalledonthenodesfordynamicallylinkedexecutables

executables descriptionlistofexecutablesloca8onspersite

physical executables

... # This is the transformation catalog. It lists information about each of the # executables that are used by the workflow. tr ls { site compute-site{ pfn "/bin/ls” type "INSTALLED” arch "x86_64" os "linux" osrelease ”centos" osversion “7" } } ...

transformation typewhetheritisinstalledor

availabletostage

mappedfromlogicaltransforma8ons

Why use Containers for your workflow?Tradi8onalwayofreferringuserexecutableinPegasus•  PegasusmatchestheaRributesofanexecutabledefinedinTransforma8onCatalogagainsttheaRributesspecifiedforsiteinSiteCatalog.

•  Thisapproachworksfineifyoursiteismadeofhomogenousnodes•  However,problemsoccurwhen

① yourunonasitewithhetrogeneousnodesandyourjoblandsonanodewhereOSisincompa8blewithyourexecutable

② Applica8onisamis-matchtothecomputenodeenvironment

•  Installlibrariesinyoursharedspaceandmakesureenvironmentreferstothoselibraries•  Needcoopera8onfromSiteAdmins.OnOSG,youcaninstallthingsinCVMFS•  TensorFlowrequiresspecificpythonlibrariesandversions.SomelibrariesmaybeeasytoinstallonlatestUbuntu,butnotonEL7

Pegasus Container Support•  IntroducedinPegasusRelease4.8

•  SupportforbothDockerandSingularity

•  UserscannowrefertocontainersintheTransforma8onCatalogwiththeirexecutablespreinstalled.

•  Userscanrefertoacontainertheywanttouse.However,theycanletPegasusstagetheirexecutabletothenode.

•  Usefulifyouwanttouseasiterecommended/standardcontainerimage.•  Usersareusinggenericimagewithexecutablestaging.

Specifying Containers in Transformation Catalog

containerReferencetothecontainertouse.

Mul8pletransforma8oncanrefertosamecontainer

image

... tr pegasus::keg{ site isi { pfn "/usr/bin/pegasus-keg arch "x86" os "linux" osrelease ”centos" osversion “7" # INSTALLED means pfn refers to path in the container. # STAGEABLE means the executable can be staged into the container type "INSTALLED" #optional attribute to specify the container to use container "centos-pegasus" } } cont centos-pegasus{ type "docker” image "docker:///centos:7" # optional site attribute to tell pegasus which site tar file # exists. useful for handling file URL's correctly image_site "optional site" # environment to be set when the job is run in the container # only env profiles are supported profile env "JAVA_HOME" "/opt/java/1.6" } ...

URLtoimageinadocker|singularityhubOR

toanexis8ngdockerimageexportedasatarfileorsingularityimage

typeCanbeeitherdockerorsingularity

Data Management for Containers

•  Userscanrefertocontainerimagesas§  DockerorSingularityHubURL’s§  DockerImageexportedasaTARfileandavailableataserver,justlikeanyotherinputdataset.

•  Ifanimageisspecifiedtoberesidinginahub

§  Theimageispulleddownasatarfileaspartofdatastage-injobsintheworkflow§  Theexportedtarfileisthenshippedwiththeworkflowandmadeavailabletothejobs§ Mo8va8on:AvoidhicngDocker/SingularityHubrepeatedlyforlargeworkflows

•  Pegasusworkerpackageisnotrequiredtobepre-installedinthecontainer§  Ifamatchingworkerpackageisnotinstalled,therequiredworkerpackageisinstalledatrun8mewhencontainerstarts

Container Execution ModelContainerizedjobsarelaunchedviaPegasusLite

•  Containerimageisputinthejobdirectoryalongwithinputdata.•  Loadsthecontainerifrequiredonthenode(applicableforDocker)•  RunascriptinthecontainerthatsetsupPegasusinthecontainerandlaunchesuserapplica8on

•  Shutdownthecontainer(applicableforDocker)•  Shipouttheoutputdatageneratedbytheapplica8on•  Cleanupthejobdirectory

•  Tradi8onalshared-fsapproachdoesnotsupportcontainers.

Directories Mounted• OnlythejobdirectorywherePegasusLiteplacestheinputsismountedinthecontainer

•  Docker–Mountedas/scratch•  Singularity–Mountedas/srv

• PegasusLiteensuresthatuserapplica8onislaunchedinthedirectorymounted

•  ConsistentwiththePegasusmodelofensuringthatuserjobislaunchedindirectorywhereit’sinputdataexists.

User Running in the Container•  Singularitycontainersalwaysruninuserspace.• Docker

•  Pegasusbeforelaunchingtheuserapplica8on•  CreatestheuserinthecontainerthatisthesameastheuserunderwhichthejobislaunchedbytheLocalResourceManagerontheremotenode

• Whydowedothis?

•  Bydefault,Dockerrunsuserapplica8onasroot•  NotrecommendedforHPCenvironment

•  Createsproblemswithstagingtheoutputscreatedinthecontainer

Reference• Documenta8on

•  hRps://pegasus.isi.edu/documenta8on/containers.php

•  Example

•  hRps://github.com/pegasus-isi/montage-workflow-v2/•  Scriptexample-dss-containers.shwillrunthemontageworkflowjobsinacontainerpulledfromthesingularityhub

Pegasus Automate,recover,anddebugscien8ficcomputa8ons.

Thank You

Questions? MatsRyngerynge@isi.edu

KaranVahi

RafaelFerreiradaSilva

RajivMayani

MatsRynge

EwaDeelman

Meet our team

est. 2001