Container networking with MidoNet

Pino de Candia, August 2015

About Midokura● Global startup with about 50+ employees● Started in 2010 with goal of building a cloud, but turned

its focus to network virtualization.● Engineering in Barcelona, Tokyo, Tel Aviv● Sales and Marketing in San Francisco● Network Virtualization for OpenStack, Docker,

Eucalyptus, and ESXi● Customers in web scale, enterprise, higher ed and

service provider segments

● Containers made easy● Daemon based● multiple backends

Docker Networkinglibcontainer(builtin):● NAT Bridge● None● Host● ContainerOverlay

NAT Bridge

Advantages:● Isolation from underlay● SimpleDrawbacks:● no easy cross-host● no advanced networking

Host Mode

Advantages:● Directly on the underlay● Full capabilitiesDrawbacks:● Lack of isolation

ContainerAdvantages:● Isolation from the underlay● Full communication containersDrawbacks:● no easy cross-host● no advanced networking

Overlay - FlannelAdvantages:● Overlay for the hosts● Each docker0 gets a routable /24● UDP and VXLAN backends● Provides configuration file for

dockerdDrawbacks:● Lack of flexibility● no advanced networking

Enter Neutron (and MidoNet)

Bring advanced networking to Containers● Neutron is the ONLY “standard” network

virtualization APIBring containers to OpenStack

Bare MetalServer

Bare MetalServer

Neutron/MidoNet transforms...

VM

VM

VM

VM VM

VM VM

VM

VM

VM

VM

VM VM

VM VM

VM

VM

VM

VM

VM VM

VM VM

VM

VM

VM

VM

VM VM

VM VM

VM

IP Fabric

Bare MetalServer

Bare MetalServer

that into this...

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM VM

VM

VM

VMVM

VM

VMVM

VM

VM

VM

VM

VM

VM

VM

FW

LB

FW

LB

Internet/WAN

FW

How to extend Docker networking?

Current way:● wrap around the docker API/cli, or● Hook to the event stream and react

Future way:● libnetwork (core or remote)

Wrapping Docker● Advanced networking as extra commands/params● Write a wrapping API● Write wrapping cli● Make sure users target the wrappers● Examples

○ Socketplane(cli)○ Weave(cli)○ Nova-docker(both)

How Nova plugs a Container’s NIC...1. the user calls nova-api to launch instance with a --nic parameter

2. nova-scheduler chooses the host and messages nova-compute

3. nova-compute delegates to the virt driver for nova: nova-docker

4. the driver pulls the docker image from glance into the local registry

5. the driver uses docker-py library to talk to docker and create the container

6. nova-compute asks the virt driver to plug and attach the nic.

7. the driver checks the port type and calls plug_ovs or plug_midonet, which creates a veth pair, puts one side in the container and one in overlay.

8. the driver sets the ip/mac config for the container side (netlink).

nova-docker wraps docker

How Nova plugs a VM’s NIC...1. the user calls nova-api to launch instance with a --nic parameter.

2. nova-scheduler chooses the host and messages nova-compute.

3. nova-compute delegates to the virt driver for libvirt

4. the driver prepares an xml definition of the VM, including the tap device needed for the port (depends on the porty type)

5. libvirt creates the VM and tap

6. the driver calls the plug method for the port type (libvirt already knows how to plug taps for MidoNet and OVS Neutron plugins).

7. The VM is created and gets the ip configuration thanks to cloud init and/or dhcp.

libnetwork (refactored libcontainer)Refactor libcontainer networking into a separate libraryExtends docker API/CLIWritten in Go by Docker for dockerMultiple drivers:

● bridge● host● null● overlay*● remote

libnetwork EntitiesDriverSandboxNetwork● Create● Delete

Endpoint● Create● Join● Leave● Delete

libnetwork “remote”

● Open to external vendors● Containerized● RPC dockerd<-->remote driver● Implements the driver entities

libnetwork neutron driver: kuryr

kuryr● Gives Swarm an overlay● Maps libnetwork entities to Neutron● Gets IPAM from Neutron● Vendor interface plugging configurable● Extra entities via labels● Opens Docker to advanced networking● Open Source and open to contributors

Contribute at https://github.com/openstack/kuryr

OpenStack Magnum