1

Tél : +33 (0)1 58 56 10 00Fax : +33 (0)1 58 56 10 01www.octo.com© OCTO 2013

50, avenue des Champs-Elysées75008 Paris - FRANCE

SDN

Follow us on Twitter!@AdrienBlind@ArnaudMazin

2

REMINDER: DOCKER NETWORKING

Agenda

INTRODUCTION TO SOFTWARE DEFINED NETWORKS

MIX UP

1

TAKE AWAY

2

4

3

3

Reminder: Docker networking

4

Host

Natively, each container runs isolated from the outside worldA bridge network (an internal, virtual switch) is provided with Docker but does not allow containerized processes to communicate together without further config (see next slides)

Isolation by design

Container ‘app’

Bridge docker0

Container ‘db’

Tomcat(port 8080)

MySQL(port 3306)

5

Exposing ports allows containers services to talk together through the bridge network

Dockerfile: add line “EXPOSE <port>” (several ports can be appended)

Command line switch: “docker run ... –expose <port>”

Example for the ‘app’ container: “docker run ... –expose 8080”

Exposing ports

Host

Container ‘app’

Bridge docker0

Container ‘db’

Tomcat(port 8080)

MySQL(port 3306)

TCP 3306 exposed

TCP 8080 exposed

« The app is able to talk to its database »

6

Linking enables a client container to get information related to a resource container (also known as the linked container)

Command line switch: “docker run -link <containername:alias>”

Example on ‘app’ container: “docker –run … –link db:dbalias”

Linking containers

Host

Container ‘app’

Bridge docker0

Container ‘db’

Tomcat(port 8080)

MySQL(port 3306)

Env variablesDBALIAS_PORT_3306_TCP=tcp://1.2.3.4:3306DBALIAS_PORT_3306_TCP_PROTO=tcp DBALIAS_PORT_3306_TCP_ADDR=1.2.3.4DBALIAS_PORT_3306_TCP_PORT=3306

« The app knows where to join its database »

7

Mapping ports enable to publish container’s ports on “external interfaces”. Ports may be translated.

Command line switch: “docker run -p <externalport:insideport>”

Example for ‘app’ container: “docker –run … –p 80:8080”

Mapping ports

Host

Container ‘app’

Bridge docker0

Container ‘db’

Tomcat(port 8080)

MySQL(port 3306)

« The app is reachable from the external network at port 80 of the host,while the database container only remains reachable to other containers »

IP_interface

TCP 80 map Tomcat’s 8080 port

External network

8

These characteristics enables containers:To either remain fully isolated or talk to all other containers of the same host having their services exposed

To map one-by-one services on the hosts interface

Hmm… What if I want to isolate groups of containers?

Conclusion

9

Introduction to Software Defined Networks

10

Traditional datacenter management

VM

VM

VM

VM

VM

VM

VM

VM

VM

Internet

Internet

DMZ

Physical overview

Logical overview

Tenant #1

Tenant #3

Tenant #2

LAN

DMZ

LAN

DMZ1

DMZ2

Logical topologies are quite different from physical ones

These designs traditionally relies a lot on low network layers (L2, etc.)

11

Multi datacenter perspective

VM

VM

VM

VM

VM

VM

VM

VM

VM

Physical overview

VM

VM

VM VM

VM

VM

VM

VM

VM

• Limited to 2048 VLANs• Lack of dynamic provisioning• Involves subnetting or encapsulation to

flow over L3 networks

Internet

12

SDNs proposes network solutions embracing cloud paradigms

Massively multi-tenantThousands tenants, massively scalable

Easy & fast (de)provisioningInfra as code, API centric

Infrastructure agnosticL3, does not stick with lower levels (physical designs, vlans & co)

Decouple infrastructure & tenants lifecycles

Cross technology, vendor agnostic

SDNs value proposal

13

Enables to abstract networking topologies & services from wiring and equipments

Centralize network intelligence

Standardized management APIs: ex. OpenFlow for both physical & virtual network equipments

SDNs overview

(Source Wikipedia)

14

On hosts (hypervisors, Docker hosts…), SDNs mostly rely on:

Allocating network bridges (virtual and internal switches)

Setting ACLs to decide which flow to allow or deny

Connecting these bridges to external world through real host’s interfaces

Host perspective

15

Network perspective – Low level isolation

Host #3

SDN network #1

SDN network #1

Host #1

SDN network #1

SDN network #1

Host #2

SDN network #1

L2-Network focusedKeep traditional paradigms but use API/centralize intelligence

Requires VLANs, VPLS… to spread same virtual networks accross several hosts

Enforces low-level isolation

Consider API-based network configuration (ie. OpenFlow) to centralize and facilitates network management, making it more dynamic

VPLS RouterPhysical net.

VPLS network

The link tags both virtual network trafics in

vlans

The VPLS ensure providing L2 networks at all ends – strong req.!

16

Full-mesh, network agnostic and encapsulated approachRelies on L3 networks and GRE, VXLAN… tunnels for inter-hosts communication (avoid using L2)

Network agnostic, til hosts can route trafic

SDN Routers must route traffic between an inner virtual net and the ext. world

Network perspective – Full mesh

Host #3

Host #1

Host #2

SDN network #1

SDN network #1

SDN network #1

SDN network #1

SDN network #1

RouterPhysical net.Flow encapsulation

L3 network

17

Mix up

18

How to support several containers related to a same tenant, accross multiple hosts (or even multiple datacenters or providers), and avoid them to talk to other containers in the same time ?

Answer to this question enables several usecasesIsolate containers of a same app without having to face limits of a single Docker host

Resilience (ex. spread an app server farm over multiple hosts)

Multi providers (ex. spread containers over several clouds/hosters, overflow mgmt…)

…

Usecases – Back to Docker

Internet

LAN

DMZ

DMZ

DMZ 1

DMZ 2

LAN

Container

Tenant #1

Tenant #3

Tenant #2

19

Host #1 Host #3Host #2

0. From where we start

Container ‘app’

Container ‘db’

Brid

ge d

ocke

r0

Container ‘app’

Container ‘db’

Brid

ge d

ocke

r0

Container ‘app’

Container ‘db’

Container ‘app’

Brid

ge d

ocke

r0

Container ‘app’

TCP ports of services running in containers are

mapped

Get rid of actual Docker bridges implementation !If not, all containers will talk together across a same host

20

Use OpenVSwitch to create bridges on each host, for each tenant’s subnet. For instance, on host #1:

» ovs-vsctl add-br tech-br» ovs-vsctl add-port tech-br tep0 -- set interface tep0 type=internal» ifconfig tep0 192.168.1.1 netmask 255.255.255.0» ovs-vsctl add-br sdn-br0» ovs-vsctl set bridge sdn-br0 stp_enable=true

1. Create SDN compliant bridges

Host #1

sdn-

br0

Host #3

sdn-

br0

sdn-

br1

Host #2

sdn-

br0

sdn-

br1

Simplified view. Detailed insight exposed in later slides

Once per host: common plumbing (detailed slide 24)

For each bridge: create and protect against ethernet loops using

Spanning Tree Protocol (beware, in complex/large deployments, it may

consumes a lot of CPU!)

21

2. Link SDN bridges

Host #1

sdn-

br0

Host #3

sdn-

br0

sdn-

br1

Host #2

sdn-

br0

sdn-

br1

Use OpenVSwitch to link corresponding bridges accross hostsIn this example, we decided to use the full-mesh approach. On host #1:

ovs-vsctl add-port sdn-br0 gre0 --set interface gre0 type=gre options:remote_ip:1.2.3.2ovs-vsctl add-port sdn-br0 gre1 --set interface gre1 type=gre options:remote_ip:1.2.3.3

gre0

gre1

1.2.3.1 1.2.3.2 1.2.3.3

Simplified view. Detailed insight exposed in later slides

22

U

3. Instanciate VMs and map them to the bridges

Host #1

sdn-

br0

Host #3

sdn-

br0

sdn-

br1

Host #2

sdn-

br0

sdn-

br1

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘db’

Instanciate containers without pre-built interfaces to avoid plugging containers to native docker0 bridge

Use “docker run … -n=false” switch in “docker run” calls

23

U

4. Connect containers to the appropriate bridge

Host #1

sdn-

br0

Host #3

sdn-

br0

sdn-

br1

Host #2

sdn-

br0

sdn-

br1

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘db’

Use pipework.sh from J. Petazzoni to easily plug containers to a chosen bridgehttps://github.com/jpetazzo/pipework

Creates network adapter in each containers, allocate them an IP (manually/static vs DHCP), and plug it to the bridge.

Per container: “pipework bridge_name $container_id container_ip/24”

24

Switch

Insight - Detailed view between two hosts

Previous drawings were simplified to ease overall understandingFollowing schema details more deeply what happened inside a single host

Host #1

sdn-br0

Container

Host #2

Container

gre0

tech-brtep0

eth0

sdn-br0

Container Container

gre0

tech-brtep0

eth0

Switch

ovs-vsctl add-br tech-br

ovs-vsctl add-port tech-br tep0 -- set interface tep0 type=internal

ovs-vsctl add-port sdn-br0 gre0 --set interface gre0 type=gre options:remote_ip:1.1.1.1

GRE tunnel in which the traffic really flows

Ethernet link between an adapter and a switch

L2 switch

Network adapter

pipework sdn-br0 $container_id 192.168.0.3/24

eth1 eth1ovs-vsctl add-br sdn-br0eth1 eth1

1.1.1.1/24 2.2.2.2/24

192.168.1.2/24

192.168.0.3/24192.168.0.2/24192.168.0.1/24 192.168.0.4/24

4

7

6

1

2

ifconfig tep0 192.168.1.1 netmask 255.255.255.0

3

L3 routed network

Docker container

Docker host

pipework sdn-br0 $container_id 192.168.0.4/24

8

192.168.1.1/24

Repeat step #6 to create additional tunnels in order to reach other hosts

ovs-vsctl set bridge sdn-br0 stp_enable=true

5Virtual, direct link establishedthrough the GRE tunnel

25

Bonus: Alternate design - OpenStack Neutron paradigmsAlternate design, based on OpenStack Neutron paradigms – notice that use of VLAN limits to 2048 tenant networks on each hosts

All VMs/containers of a same tenant network are segregated inside a dedicated, local VLAN of a shared unique bridge

Full-mesh of GRE tunnels between all hosts

On each host, local mapping between a local tenant network VLAN and its GRE identifier shared across all hosts

Full story from VM A to B (tenant #1): traffic outgoing A is tagged with VLAN 20 entering br-int, flows to br-tun, is untagged entering GRE tunnel while the GRE identifier 11111 is set on transmitted packets. At the other end of the GRE tunnel, the traffic having the GRE id 11111 is assigned to VLAN 40, then flow outside br-tun, to br-int, and is finally untagged before entering B.

Full story from A to C (tenant #1): traffic is tagged with VLAN 20 entering br-int, then flows to br-eth1 which finally untag inbound trafic (or assign a different VLAN corresponding to the external world)

Switch

Host #1

br-int

br-eth0

eth0

Switch

GRE tunnel in which the traffic really flows

Ethernet link between an adapter and a switch

L2 switch

Network adapterL3 routed network

VM or container

Host/server

br-eth1

eth1

br-tun

Tenant #2Local VLAN 30

Tenant #1Local VLAN 20

A single bridge is used for all VMs/containers ; VMs of different tenants isolated using local VLANs (not exposed externally !)

A single bridge is used as end-points for all GRE tunnels used for full-mesh

One bridge is created for each physical interface

Host #2

br-int

br-eth0

eth0

br-tun

Tenant #2Local VLAN 50

Tenant #1Local VLAN 40

Switch or VLAN (not related to

internal VLANs)

gre0

gre0

A B

ServerC

Flow rules:Tenant #1: VLAN 20 GRE ID 11111Tenant #2: VLAN 30 GRE ID 11112

Single patch link between the two bridges supporting all traffic from/to full-mesh

Flow rules:Tenant #1: VLAN 40 GRE ID 11111Tenant #2: VLAN 50 GRE ID 11112

26

Take away

27

Use Docker for containers hosting, externalize SDN managementDisable bridges management features in Docker, use OpenVSwitch

Abstract from low level network considerations whenever possible, between hosts (L2 VLANs for instances): consider tunneling

Get furtherUse OpenStack Neutron to centralize & automatize the whole network conf.

You definitively use VLANs ? Consider encapsulating several VLANs in your own tenant network

Other (dirty) options:Docker in VMs nesting

Multiple Docker instances on the same host

ebtables/iptables/openvswitch acls on flat network

Take away

28

?