Supporting Docker inEmulab-Based Network Testbeds
DavidJohnson,ElijahGrubb,EricEideUniversityofUtah
�2
�2
�2
�2
• overthecourseofastudy…
• prototypeonlaptop
• networktestbed
• commercialcloud
• needtomoveexperimental artifactsaround
�3
• overthecourseofastudy…
• prototypeonlaptop
• networktestbed
• commercialcloud
• needtomoveexperimental artifactsaround
�3
• overthecourseofastudy…
• prototypeonlaptop
• networktestbed
• commercialcloud
• needtomoveexperimental artifactsaround
�3
• overthecourseofastudy…
• prototypeonlaptop
• networktestbed
• commercialcloud
• needtomoveexperimental artifactsaround
�3
• overthecourseofastudy…
• prototypeonlaptop
• networktestbed
• commercialcloud
• needtomoveexperimental artifactsaround
�3
• overthecourseofastudy…
• prototypeonlaptop
• networktestbed
• commercialcloud
• needtomoveexperimental artifactsaround
�3
This talk
• extendedEmulabsouserscancreateexperimentsinwhichsomeorallnodesareDockercontainers
• challenges• preservingusers’“testbedexperience”• meshingwithEmulab’sinfrastructure
• results• justworks:52/60topDockerHubimagesautomaticallyadapted• supportslarge(5K-node)experiments
�4
Docker
�5
Docker
• basedoncontainers
• filesystemspopulatedviaimages
�5
PhysicalHostHostOSDocker
Container
Filesystem
App
Docker
• basedoncontainers
• filesystemspopulatedviaimages
• imagescreatedviaDockerfiles
�5
PhysicalHostHostOSDocker
Container
Filesystem
App
ImageDockerfile
Emulab
• testbedmanagementsoftware
• allocatesphysicaland virtualresourcestousers
• configuresresources
• isolatesusersfromeach other
�6
Emulab
�7
Emulab
• organizedaroundprofiles
• profilesareinstantiated tomakeexperiments
�7
Profile
Emulab
• organizedaroundprofiles
• profilesareinstantiated tomakeexperiments
• nodes’diskspopulatedviadiskimages
�7
Profile
Emulab
• organizedaroundprofiles
• profilesareinstantiated tomakeexperiments
• nodes’diskspopulatedviadiskimages
• in-experimentservices
�7
Profile Diskimages
Goal: Emulab + Docker should “just work”
�8
Goal: Emulab + Docker should “just work”
• containersinEmulabarejustanotherkind ofvirtualnode
�8
Dockerimages
Goal: Emulab + Docker should “just work”
• containersinEmulabarejustanotherkind ofvirtualnode• EmulabusercanchooseanyDockerimage
• preserveEmulab’sexperimenterservices• e.g.,SSH,local/remotestorageaccess,…
�8
Dockerimages
Goal: Emulab + Docker should “just work”
• containersinEmulabarejustanotherkind ofvirtualnode• EmulabusercanchooseanyDockerimage
• preserveEmulab’sexperimenterservices• e.g.,SSH,local/remotestorageaccess,…
• preserveEmulab’snetworkservices• e.g.,controlnetwork,trafficshaping,…
�8
Dockerimages
Goal: Emulab + Docker should “just work”
• containersinEmulabarejustanotherkind ofvirtualnode• EmulabusercanchooseanyDockerimage
• preserveEmulab’sexperimenterservices• e.g.,SSH,local/remotestorageaccess,…
• preserveEmulab’snetworkservices• e.g.,controlnetwork,trafficshaping,…
• preserveDockeruserexperience• e.g.,“dockercommit”
�8
Dockerimages
�9
httpd:latest
Preserving Emulab’s experimenter services
• shellaccesstonodes
• remoteandlocalstorage
• networkconfiguration• addressing,routing,shaping
• startupprograms
�10
Preserving Emulab’s experimenter services
• shellaccesstonodes
• remoteandlocalstorage
• networkconfiguration• addressing,routing,shaping
• startupprograms
• typicalDockerimagesareminimalappliances
• runtheapplicationonly• notpreparedtohostotherservices
�10
httpd:latest
augmentation
generateanewDockerfile,startingfromtheuser’schosenimage,
andaddingtestbedsoftware
Augment the startup
�12
httpd:latest
Augment the startup
• maketemporarycontainer
�12
httpd:latest
Container
Augment the startup
• maketemporarycontainer• addbuildtoolchain
�12
httpd:latest
Containerbuildtools
Augment the startup
• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit
�12
httpd:latest
Containerbuildtools
runit
Augment the startup
• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit
�12
httpd:latest
Containerbuildtools
runit
runit
Augment the startup
• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit
�12
httpd:latest
runit
Augment the startup
• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit
�12
httpd:latest
runit
FROMhttpd:latest
COPY…runit…RUN…runit-setup…
Augment the startup
• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit• addDockerfileinstructionstoinstallrunit
�12
httpd:latest
runit
FROMhttpd:latest
COPY…runit…RUN…runit-setup…
Augment the startup
• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit• addDockerfileinstructionstoinstallrunit• configurerunittoruntheoriginalENTRYPOINT
�12
httpd:latest
runit
FROMhttpd:latest
COPY…runit…RUN…runit-setup…
Augment the startup
• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit• addDockerfileinstructionstoinstallrunit• configurerunittoruntheoriginalENTRYPOINT• whenaugmentedimageisused,setENTRYPOINTtorunit
�12
httpd:latest
runit
FROMhttpd:latest
COPY…runit…RUN…runit-setup…
Add the Emulab “client-side” software
• maketemporarycontainer• compileandpackageEmulabclient-sidesoftware• addDockerfileinstructionstoinstallthesoftware
• user-selectablelevelsofaugmentation
�13
FROMhttpd:latest
COPY…RUN…runit-setup…&&…emulab-setup…
Local registry
�14
Dockerregistry
Local registry
• cacheaugmentedimagesinatestbed-localDockerregistry
• speedssubsequentexperimentcreation
• integratedwithEmulab’suserauthentication&authorizationmodel
�14
Dockerregistry
httpd:latest perl:5.28
redis:4.0 node:8.11.3
mysql:5.7 erlang:21.0
Preserving Emulab’s network services
• separatecontrolnetwork
• experimenttrafficshaping
• control-networkfirewalls
• DNS
�15
Preserving Emulab’s network services
• separatecontrolnetwork
• experimenttrafficshaping
• control-networkfirewalls
• DNS
• Docker’sContainerNetworkModel(CNM)ismismatchedtodemandsofanetworktestbed
• tooabstract
• triestocontroltoomuch
• missingfeatures
�15
leverage the physical host
managenetworkservicesonthephysical-hostsideof
containers’virtualnetworkinterfaces
Control network
�17
PhysicalHost
physicalcontrolnetwork
Control network
• atphysical-hostboot• createdockercnetvirtualnetwork
• bridgetothephysicalcontrolnetwork
�17
PhysicalHost
dockercnet
physicalcontrolnetwork
Control network
• atphysical-hostboot• createdockercnetvirtualnetwork
• bridgetothephysicalcontrolnetwork
• atcontainerstartup• connecttodockercnet• setupNATtoexposeSSHoverthephysicalhost’spublicIPaddress
�17
PhysicalHost
Container Container Container
dockercnet
physicalcontrolnetwork
Traffic shaping and firewalls
�18
PhysicalHost
dockercnet
controlnetwork
experimentnetworks
Traffic shaping and firewalls
• Emulabsubscribestolife-cycleeventsofeachcontainer
• atcontainerstartup• installtcrulesforexpt.-networktrafficshaping
• installiptablesrulesforcontrol-networkfirewalling
�18
PhysicalHost
dockercnet
controlnetwork
experimentnetworks
Traffic shaping and firewalls
• Emulabsubscribestolife-cycleeventsofeachcontainer
• atcontainerstartup• installtcrulesforexpt.-networktrafficshaping
• installiptablesrulesforcontrol-networkfirewalling
• atcontainershutdown• removetherules
�18
PhysicalHost
Container Container Container
dockercnet
controlnetwork
experimentnetworks
firewall
shaping
Dedicated and shared modes
�19
PhysicalHost
Container Container Container
experimentnetworks
Dedicated and shared modes
• dedicated—containersrunonphysicalmachinereservedtooneexperiment
�19
PhysicalHost
Experiment
Container Container Container
experimentnetworks
Dedicated and shared modes
• dedicated—containersrunonphysicalmachinereservedtooneexperiment• shared—physicalmachinemayhostcontainersfromseveralexperiments
�19
PhysicalHost
Expt.1 Expt.2
Container Container Container
experimentnetworks
Dedicated and shared modes
• dedicated—containersrunonphysicalmachinereservedtooneexperiment• shared—physicalmachinemayhostcontainersfromseveralexperiments
�19
PhysicalHost
Expt.1 Expt.2
Container Container Container
experimentnetworks
192.168
.1.1
192.168
.1.1
Dedicated and shared modes
• dedicated—containersrunonphysicalmachinereservedtooneexperiment• shared—physicalmachinemayhostcontainersfromseveralexperiments
• wemodifiedDockertosupportmultiple,isolatedlayer2netsonasinglephysicalhost
�19
PhysicalHost
Expt.1 Expt.2
Container Container Container
experimentnetworks
192.168
.1.1
192.168
.1.1
Implemented & deployed
• supportedOSes• AlpineLinux3.6,3.7,3.8• CentOS7• Debian8,9,sid• Ubuntu14.04,16.04,18.04
• registriesat
�20
…
evaluation•60mostpopularimagesfromDockerHub
•fourresearchDockerimages•timetoaugmentDockerimages•timetocreatelargeexperiments
�22
Category DockerImages
Linuxdistro alpine,centos,debian,ubuntu,amazonlinux,busybox,fedora
Debian buildpack-deps,cassandra,chronograf,drupal,elasticsearch,ghost,golang,gradle,groovy,haproxy,httpd,influxdb,java,jenkins,jruby,kibana,logstash,mariadb,maven,memcached,mongo,mysql,nextcloud,nginx,node,openjdk,owncloud,percona,perl,php,postgres,python,rabbitmq,redis,rethinkdb,rocket.chat,ruby,sentry,solr,sonarqube,tomcat,wordpress,telegraf
Alpine consul,docker,kong,neo4j,vault,registry
Scratch hello-world,nats,swarm,traefik
�22
Category DockerImages
Linuxdistro alpine,centos,debian,ubuntu,amazonlinux,busybox,fedora
Debian buildpack-deps,cassandra,chronograf,drupal,elasticsearch,ghost,golang,gradle,groovy,haproxy,httpd,influxdb,java,jenkins,jruby,kibana,logstash,mariadb,maven,memcached,mongo,mysql,nextcloud,nginx,node,openjdk,owncloud,percona,perl,php,postgres,python,rabbitmq,redis,rethinkdb,rocket.chat,ruby,sentry,solr,sonarqube,tomcat,wordpress,telegraf
Alpine consul,docker,kong,neo4j,vault,registry
Scratch hello-world,nats,swarm,traefik
fullysupportedpartiallysupportednotsupported
�23
Category DockerImages
Linuxdistro alpine,centos,debian,ubuntu,amazonlinux,busybox,fedora
Debian buildpack-deps,cassandra,chronograf,drupal,elasticsearch,ghost,golang,gradle,groovy,haproxy,httpd,influxdb,java,jenkins,jruby,kibana,logstash,mariadb,maven,memcached,mongo,mysql,nextcloud,nginx,node,openjdk,owncloud,percona,perl,php,postgres,python,rabbitmq,redis,rethinkdb,rocket.chat,ruby,sentry,solr,sonarqube,tomcat,wordpress,telegraf
Alpine consul,docker,kong,neo4j,vault,registry
Scratch hello-world,nats,swarm,traefik
fullysupportedpartiallysupportednotsupported
�23
Category DockerImages
Linuxdistro alpine,centos,debian,ubuntu,amazonlinux,busybox,fedora
Debian buildpack-deps,cassandra,chronograf,drupal,elasticsearch,ghost,golang,gradle,groovy,haproxy,httpd,influxdb,java,jenkins,jruby,kibana,logstash,mariadb,maven,memcached,mongo,mysql,nextcloud,nginx,node,openjdk,owncloud,percona,perl,php,postgres,python,rabbitmq,redis,rethinkdb,rocket.chat,ruby,sentry,solr,sonarqube,tomcat,wordpress,telegraf
Alpine consul,docker,kong,neo4j,vault,registry
Scratch hello-world,nats,swarm,traefik
fullysupportedpartiallysupportednotsupported
Emulabautomaticallyadapted52/60imagesintothetestbedenvironmentandinstantiatedcontainersfromthem.
Scalability
• createlargeexperimentswithDockercontainers
• ineachtrial• 200containersperphysicalhost• eachcontainerrunsaugmentedubuntu:14.04imagefromtestbed’slocalregistry
• allcontainersattachedtoaLAN
• physicalhosts:CloudLabxl170nodesrunningUbuntu16.04
�24
Scalability
• createlargeexperimentswithDockercontainers
• ineachtrial• 200containersperphysicalhost• eachcontainerrunsaugmentedubuntu:14.04imagefromtestbed’slocalregistry
• allcontainersattachedtoaLAN
• physicalhosts:CloudLabxl170nodesrunningUbuntu16.04
• 1–25physicalhosts• yielding200–5,000containers
�24
Scalability
• createlargeexperimentswithDockercontainers
• ineachtrial• 200containersperphysicalhost• eachcontainerrunsaugmentedubuntu:14.04imagefromtestbed’slocalregistry
• allcontainersattachedtoaLAN
• physicalhosts:CloudLabxl170nodesrunningUbuntu16.04
• 1–25physicalhosts• yielding200–5,000containers
• measure• elapsedtimetofirstcontainer• avg.creationtimeforeachcontainerafterthefirst
• elapsedtimetocreateallcontainersoneachphysicalhost
• elapsedtimetocreatefullexpt.
�24
Scalability
• createlargeexperimentswithDockercontainers
• ineachtrial• 200containersperphysicalhost• eachcontainerrunsaugmentedubuntu:14.04imagefromtestbed’slocalregistry
• allcontainersattachedtoaLAN
• physicalhosts:CloudLabxl170nodesrunningUbuntu16.04
• 1–25physicalhosts• yielding200–5,000containers
• measure• elapsedtimetofirstcontainer• avg.creationtimeforeachcontainerafterthefirst
• elapsedtimetocreateallcontainersoneachphysicalhost
• elapsedtimetocreatefullexpt.
• repeateachtrial3×,reportavgs.
�24
�25
�25
200containers 14minutes
�25
200containers 14minutes
5,000containers 1.87hours
�25
200containers 14minutes
5,000containers 1.87hours
Conclusion:acceptableperformance,butmoreserver-sideoptimizationwillbeneededforlargeexperiments.
�25
parallelized
�25
one-timesetup
Conclusion
• Emulab+Docker“justworks”• experimenterservices—automaticaugmentation• networkservices—physicalhostcontrol&minorDockermods
• supportsexistingDockerimages• promotesartifactportability• promotesresearchrepeatability
• availableinEmulab-basedtestbedsnow!
�26
EricEidewww.cs.utah.edu/~eeide/email:[email protected]:@eeide