Container Networking and Openstack Fernando Sanchez - @fernandosanchez Fawad Khaliq - @fawadkhaliq March, 2016
Copyright © PLUMgrid, Inc. 2011-2015
Agenda
Containers, Orchestration and Networking
• Containers: runtimes
• Container Orchestration tools
• Container Networking models
• Early (host, bridge, container modes)
• Modern (CNM, CNI)
Containers and Openstack
• Magnum
• Kuryr PLUMgrid's involvement in the container ecosystem
Containers, Orchestration and Networking
Copyright © PLUMgrid, Inc. 2011-2015
Containers: runtimes
• LXC - Linux standard • Docker - Docker
• Rocket - CoreOS • Drawbridge – Microsoft • LXD - Canonical (uses LXC) • And also an open container project: runC
Copyright © PLUMgrid, Inc. 2011-2015
Containers: Orchestration
• Swarm (plus Docker Datacenter) - Docker
• Kubernetes - Google
• Mesos - Berkeley, Airbnb, Twitter, Apple…. Mesosphere
• Fleet, Tectonic - CoreOS
• Rancher – Rancher Labs • others…
Copyright © PLUMgrid, Inc. 2011-2015
Early: • Bridge mode • Host mode • Container mode
Evolution: • Container Networking Model (CNM) • Container Networking Interface (CNI)
Container Networking strategies
Copyright © PLUMgrid, Inc. 2011-2015
Bridge-mode Networking
• A private “docker0” bridge is provisioned on the host
• A private namespace for each container is provisioned inside that bridge
• Containers’ ethX are mapped to private bridge interfaces
• NAT (w/Iptables) is used to map between each private container and the host’s public interface
Container networking(I)
Copyright © PLUMgrid, Inc. 2011-2015
Bridge-mode Networking
• Each container needs a port mapped from eth0 into their private interface
• IPtables keeps a NAT session
in the conntrack per each container mapping public/private
• Different ports required on public per container
• Default mode of operation
• Use of NAT à Performance? Scalability? Operations? Orchestration?
Container networking(I)
Copyright © PLUMgrid, Inc. 2011-2015
Host-mode Networking
• Docker reuses the networking namespace of the main host (it’s virtually “the same interface” for the host and the container)
• The only way to share the interface is to use different ports per container.
• Recommended for production in Docker
• Faster… but secure?
Container networking (II)
Copyright © PLUMgrid, Inc. 2011-2015
Container-mode Networking
• Docker reuses the networking namespace of another container.
• Typically, create a container with bridge mode then create a set of containers sharing the interface with the first container
• The only way to share the interface is to use different ports per container.
• Kubernetes used work like this, where a set of containers sharing the bridge is a POD
• Recently moved to CNI (see later)
• On the way out?
Container networking (III)
eth0 iptables
vethxxx
docker0
vethxxx
LO
eth0
Docker host 1.2.3.4:80
172.17.0.17:49154
LO
eth0
172.17.0.17:49155
Nginx container 1 Nginx container 2
Copyright © PLUMgrid, Inc. 2011-2015
libnetwork Docker’s interface between the docker daemon and the network
Docker
Remote drivers (Weave, PLUMgrid, Kuryr, etc.)
libnetwork
Native drivers (Null, Bridge, Overlay)
Copyright © PLUMgrid, Inc. 2011-2015
• Network Sandbox = Isolated environment for the container’s network config…. (similar to a “network namespace”)
• Endpoint = Interface • Network = Group of endpoints
Container networking (IV) CNM : Container Networking Model (Docker)
Copyright © PLUMgrid, Inc. 2011-2015
docker network
create Create a network
connect Connect container to a network
disconnect Disconnect container from a network
inspect Display network information
ls List all networks
rm Remove a network
Docker network CLI
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network create \
—driver overlay \
—subnet=192.168.1.0/24 \
test-net
Docker: create a network
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network ls
NETWORK ID NAME DRIVER
a0ee12ceb7ed test-net overlay
Docker: list networks
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network connect \
test-net test-cont
$ docker run \
—itd —name=test-cont \
—net=test-net test-image
Docker: connect a network
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network disconnect \
test-net test-cont
Docker: disconnect a network
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network delete test-net
Docker: delete a network
Copyright © PLUMgrid, Inc. 2011-2015
Treats container / pod of containers synonymous to Linux network namespaces
Networks described on JSON based format for network and IPAM config in /etc/cni/net.d {
"name": “test-net", "type": ”bridge", #type of network/plugin bridge,macvlan,ipvlan,
COMMERCIAL "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", #can be used for pluggable ipams "subnet": "10.22.0.0/16", "routes": [ { "dst": "0.0.0.0/0" } ] } }
Plugin supports two commands: add/remove container to/from network
Kubernetes has already adopted CNI and code went in for initial draft recently.
Container networking modes (IV) CNI : Container Networking Interface (CoreOS, Kubernetes)
Copyright © PLUMgrid, Inc. 2011-2015
• Magnum: provides orchestration using Swarm, Mesos and Kubernetes
• Kuryr: provides networking for containers using Neutron as a backend.
Containers and container networking in OpenStack
Copyright © PLUMgrid, Inc. 2011-2015
• Provides container orchestration from an Openstack API, abstracting the underlying Container Orchestration Engines (Swarm, Kubernetes, Mesos).
• Uses Heat, Nova, Neutron, user-data, cloud-init.
• Uses Nova instances on Neutron networks to launch container orchestration engines (COEs)
• Containers are launched inside Nova instances. Support for bare metal container is not there yet.
• Use non-Neutron networking options for container right now.
Magnum: Container-as-a-service
Copyright © PLUMgrid, Inc. 2011-2015
• Works as a proxy for libnetwork to provide networking for Docker using Neutron as a backend.
• Plans to support both CNM and CNI. • Kubernetes integration in discussion • Is not integrated with Magnum yet. • Kuryr is currently a remote driver for libnetwork.
Kuryr: Networking for containers with OpenStack
Copyright © PLUMgrid, Inc. 2011-2015
Kuryr: Networking for containers with OpenStack
PLUMgrid’s involvement in the container ecosystem
Copyright © PLUMgrid, Inc. 2011-2015
• Remote driver for libnetwork • Available at:
• https://github.com/plumgrid/libnetwork-plugin • Uses PLUMgrid’s Open Networking suite
PLUMgrid libnetwork plugin
Copyright © PLUMgrid, Inc. 2011-2015
PLUMgrid libnetwork plugin
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network create
PLUMgrid libnetwork plugin
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network connect
PLUMgrid libnetwork plugin
Copyright © PLUMgrid, Inc. 2011-2015
$ docker network connect
PLUMgrid libnetwork plugin
Demo of PLUMgrid libnetwork plugin and integration with Swarm
Copyright © PLUMgrid, Inc. 2011-2015
• PLUMgrid is an active member of the Kuryr development community.
• Working on the nested container networking design in the OpenStack community. This will bring Neutron networking to Magnum containers.
• Design specification can be found here
• https://review.openstack.org/#/c/269039/
Kuryr, Magnum, Neutron Integration: Nested container
Copyright © PLUMgrid, Inc. 2011-2015
VM1
c1 c2
VM2
c1 c2
router1
vm-network2 vm-network1
cnt-network2 cnt-network1
What’s the nested container problem?
Copyright © PLUMgrid, Inc. 2011-2015
• https://docs.docker.com/engine/userguide/networking/work-with-networks/
• https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/network/cni
• https://docs.docker.com/engine/userguide/networking/get-started-overlay/
• http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-libnetwork.html
• http://mail-archives.apache.org/mod_mbox/mesos-issues/201601.mbox/%3CJIRA.12910446.1446664866000.159860.1453306119973@Atlassian.JIRA%3E
• https://github.com/kubernetes/kubernetes/search?utf8=%E2%9C%93&q=cni
References