Docker Networking Tutorial– Basic Options and

Access Control

Srini Seetharamansrini@sdnhub.org

November, 2014

Key Takeaways

1. Docker networking is in early stage and diverse

2. Applications must choose what networking is right for their needs. It is possible to use same principles as VMs

3. Open vSwitch brings powerful networking capabilities

4. LorisPack is an easy way to add pod-level isolation for Docker containers

5. User space vs Kernel space packet processing is an important design choice

Copyright Reserved

• Over the past few years, LXC came up as an alternative to VM for running workload on hosts

• Each container is a clone of the host OS

• Docker brought Linux containers to prominence

‒ Tracks application configuration and possibly archives to DockerHub

Linux Containers

3

Container 1

App X

Container 2 Container 3

Host OS

Guest root

App Y

Guest root

App Z

Guest root

Copyright Reserved

Docker

• Excellent way to track application dependencies and configuration in a portable format.

• For instance, the Dockerfileon the right can be used to spawn a container with nginxLB and accessed at a host port

$ docker build XYZ

$ docker images

$ docker run --name=nginx1 -d –i nginx

4

# Pull base image.

FROM dockerfile/ubuntu

# Install Nginx.

RUN \

add-apt-repository -y ppa:nginx/stable && \

apt-get update && \

apt-get install -y nginx && \

rm -rf /var/lib/apt/lists/* && \

echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \

chown -R www-data:www-data /var/lib/nginx

# Define mountable directories.

VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs", "/etc/nginx/conf.d", "/var/log/nginx"]

# Define working directory.

WORKDIR /etc/nginx

# Define default command.

CMD ["nginx"]

# Expose ports.

EXPOSE 80

EXPOSE 443

Copyright Reserved

Setup: 3 containers on 1 host

Spawn 1 nginx and 2 bash containers in detached mode

$ docker run –i –d --name=nginx1 nginx

$ docker run -i -d --name=c1 base /bin/bash

$ docker run -i -d --name=c2 base /bin/bash

List running containers Linux bridge ports

$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

4846cacc4a1e nginx:latest "nginx -g" 4 days ago Up 4 days 443/tcp,

80/tcp nginx1

0ad1a5eeff69 base:latest "/bin/bash" 4 days ago Up 4 days c1

a51357948bfd base:latest "/bin/bash" 4 days ago Up 4 days c2

5

Copyright Reserved

High-level Concepts

Namespace Containerized networking at the process level managed at /proc

Linux Bridge L2/MAC learning switch built into the Kernel to use for forwarding

OpenvSwitch Advanced bridge that is programmable and supports tunneling

NAT Network address translators are intermediate entities that translate IP address + Ports (Types: SNAT, DNAT)

iptables Policy engine in kernel that is used for managing packet forwarding, firewall, NAT features

Unix domain sockets

File descriptor based communication that is restricted to a single host. Works like a FIFO pipe.

User-space vs Kernel-space

Application domain that regulates access to resources and performance possible. • Container applications run in user-space• Typically network forwarding runs in kernel space

Options for Docker Networking

7

Copyright Reserved

"NetworkSettings": {"Bridge": "docker0","Gateway": "172.17.42.1","IPAddress": "172.17.0.19","IPPrefixLen": 16,"MacAddress": "02:42:ac:11:00:0f","PortMapping": null,"Ports": {

"443/tcp": [{

"HostIp": "0.0.0.0","HostPort": "49157"

}],"80/tcp": [

{"HostIp": "0.0.0.0","HostPort": "49158"

}]

}

Networking Still in Early Stages

• Today Docker usage is predominantly within a single laptop or host.

• The default network assigned is a port on Linux bridge docker0

• The network on right is allocated to the nginx container we spawned.

8

Copyright Reserved

Many ways to network in Docker

• Many of these are similar to what we can do with VM(except the Unix-domain socket method of direct access)

9

Host

Container

CContainer D Container E Container FContainer A Container B

Direct

Host

network

Unix-domain

sockets and

other IPC

Docker0

Linux bridge

Docker proxy

(using iptables)

Open vSwitch

Port

mapping

Copyright Reserved

Option 1: Docker0 bridge

• Default network automatically created when no additional options “--net“ or “-P” are specified

• Each container is addressed by a static IP address assigned by Docker

• Similar to what we have as default with KVM or VirtualBox

• Host can reach container with IP on the bridge

• But, outside traffic cannot reach the container

10

Host

Nginx1172.17.0.18

C1172.17.0.19

C2172.17.0.20

172.17.42.1

Docker0 bridge

eth0 eth0 eth0

veth002aa7a veth6df8377 veth7b0e4c6

eth0192.168.50.16

Copyright Reserved

Option 1: Docker0 bridge

Check Linux bridge ports and NAT rules under the hood

# iptables –L –t nat -n

...

Chain POSTROUTING (policy ACCEPT)

target prot opt source destination

MASQUERADE all -- 172.17.0.0/16 anywhere

# sudo brctl show

bridge name bridge id STP enabled interfaces

docker0 8000.56847afe9799 no veth002aa7a

veth6df8377

veth7b0e4c6

# docker inspect --format='{{.NetworkSettings}}' nginx1

(See for yourself)

11

Copyright Reserved

Option 2: Port mapping

• Provide access to the container from outside byallocating a DNAT port in the range 49153-65535

• Still uses Linux bridge docker0, butadds iptables rules for the DNAT

• In our example, nginx2 container is reachable byaccessing 192.168.50.16:49155

# docker run -P -d -i --name=nginx2 -t nginx

# iptables –L –t nat -n

...

Chain DOCKER (2 references)

target prot opt source destination

DNAT tcp -- anywhere anywhere tcp dpt:49155 to:172.17.0.19:80

DNAT tcp -- anywhere anywhere tcp dpt:49156 to:172.17.0.19:443

...

12

Host

nginx2 c1

172.17.42.1

Docker0 bridge

eth0 eth0

veth79ed06d veth6df8377

eth0192.168.50.16

Copyright Reserved

For the new nginx2 container, we show network settings below# docker inspect nginx2

"NetworkSettings": {"Bridge": "docker0","Gateway": "172.17.42.1","IPAddress": "172.17.0.19","IPPrefixLen": 16,"MacAddress": "02:42:ac:11:00:0f","PortMapping": null,"Ports": {

"443/tcp": [{

"HostIp": "0.0.0.0","HostPort": "49157"

}],"80/tcp": [

{"HostIp": "0.0.0.0","HostPort": "49158"

}]}}

Option 2: Port mapping

Advanced:‒ It is possible to restrict the port

mapping to listen on specific host IP address and/or a specific host port number

‒ Use -p option as follows

# docker run \–p host_IP:host_port:container_port\–d –i –t nginx

13

Copyright Reserved

Option 3: Host Option 4: Container

14

Give full access of the host network to the container using --net=host option# docker run --net=host

--name=c3 -i –d –t base /bin/bash

Check network within container using ifconfig command through exec# docker exec c3 ifconfig eth0

eth0 Link encap:EthernetHWaddr 52:54:00:0d:3c:9finet addr:192.168.50.16Bcast:192.168.50.255

Host can talk to container using lo (localhost) interface

Containers can listen on privileged ports (i.e., port numbers < 1024) of host

Give full access to network of a container XX to the new container YY using --net=container:XX option# docker run --net=container:nginx1

--name=c4 -i –d –t base /bin/bash

Check network within container using ifconfig command through exec# docker exec c4 ifconfig eth0

eth0 Link encap:EthernetHWaddr 02:42:ac:11:00:12inet addr:172.17.0.18Bcast:0.0.0.0

Container XX can talk to container YY using lo (localhost) interface

Copyright Reserved

Option 5: Open vSwitch (OVS)

• Similar to Linux bridge, but different technology

‒ Today, this is not the default with Docker

‒ Allows programming with OVSDB and OpenFlow protocols

• Why? OpenvSwitch has many useful features!

‒ VxLAN, GRE, VLAN based encapsulation and L2 forwarding

‒ Encapsulation allows containers to pick any MAC/IP they want

‒ Also possible to do L3 routing, ARP proxy etc, load-balancing

‒ Access control, traffic rate limiting and prioritization

‒ 10G/s or more packet processing throughput possible

‒ 1) kernel, or 2) userspace, with optionally DPDK acceleration

15

Copyright Reserved

Option 5: Open vSwitch (OVS)

Usage steps

1. Create OVS bridge and assign a gateway interface

2. Use --net=none when spawning container

3. Manually connect containers to the OVS bridge using veth pair

16

# ovs-vsctl add-br br0

# ovs-vsctl add-port br0 ovsbr0p1 -- set interface ovsbr0p1 type=internal

# ifconfig ovsbr0p1 192.168.50.1 netmask 255.255.255.0 up

# iptables -t nat -A POSTROUTING -s 192.168.50.0/24 -j MASQUERADE

# docker run --name=c1 --net=none -d -i -t base /bin/bash

Docker working towards allowing plugin

mechanism for integrating automation agents

Copyright Reserved

Option 5: Open vSwitch (OVS)

• Extract process ID for this container, and create network namespace for it

• Create veth device pair and move one to the processes’ network namespace

• Rename that device to eth0 and assign IP address + route to it

• Add pair interface to the OVS bridge

17

# pid=`docker inspect --format '{{ .State.Pid }}' $1`

# ln -s /proc/$pid/ns/net /var/run/netns/c1

# ip netns exec c1 ip link set dev peertapc1 name eth0

# ip netns exec c1 ip link set eth0 up

# ip netns exec c1 ip addr add 172.27.0.2/24 dev eth0

# ip netns exec c1 ip route add default via 172.27.0.1

# ip link add tapc1 type veth peer name peertapc1

# ip link set peertapc1 netns c1

# ifconfig tapc1 up

# ovs-vsctl add-port br0 tapc1

Copyright Reserved

Option 6: Unix-domain socket or pipe

• All inter-process communication mechanisms can be used with containers, especially Unix domain sockets or pipes. For instance,

• Apps on c1 and c2 can communicate over following Unix socket address

18

# docker run --name=c1 -v /var/run/foo:/var/run/foo -d -i -t base /bin/bash

# docker run --name=c2 -v /var/run/foo:/var/run/foo -d -i -t base /bin/bash

bind(socket_fd, (struct sockaddr *) &address,sizeof(struct sockaddr_un));

listen(socket_fd, 5);

while((connection_fd = accept(socket_fd,(struct sockaddr *) &address, &address_length)) > -1)

nbytes = read(connection_fd, buffer, 256);

struct sockaddr_un address;

address.sun_family = AF_UNIX;

snprintf(address.sun_path, UNIX_PATH_MAX, "/var/run/foo/bar");

connect(socket_fd, (struct sockaddr *) &address,sizeof(struct sockaddr_un));

write(socket_fd, buffer, nbytes);

C1 : Server.c C2 : Client.c

Other Parameters

19

Copyright Reserved

Other Networking Parameters

These are option assigned to the Docker daemon during startup

• Change default Linux bridge using --bridge

‒ IP address needs to be set for the bridge using --bip

• Change range of fixed IPs assigned to the container‒ Typical default is the same range as that assigned to bridge with --bip

• Change default MTU of the containers using --mtu option‒ The actual MTU will be the least of the uplink interface, and this mtu

• Set DNS server and domain using --dns, --dns-search options

• --ip-forward enables proxy forwarding at the bridge

• --iptables enable using iptables for access control

20

Securing containers using iptables

… default DROP policy

21

Copyright Reserved

Access Control using icc and links

Tighter access control on communication between containers

1. Start docker daemon with --icc=false option. This inserts a default DROP rule in iptables thereby preventing any spawned containers from communicating with each other.

22

$ sudo iptables –L –n...Chain FORWARD (policy ACCEPT)target prot opt source destinationACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHEDACCEPT all -- 0.0.0.0/0 0.0.0.0/0DROP all -- 0.0.0.0/0 0.0.0.0/0

Copyright Reserved

Access Control using icc and links

Tighter access control on communication between containers

2. Connect two containers using the --link option in docker run

‒ Some environment variables (e.g., NGINX1_PORT) are set in target container and entry in the /etc/hosts file for easy access to source

23

nginx1c1

Docker0 bridge

c2

X

$ docker run --name=nginx1 -P -d -i -t nginx$ docker run --name=c1 -d -i -t --link nginx1:nginx1 base /bin/bash$ docker run --name=c2 -d -i -t base /bin/bash

$ docker exec c1 cat /etc/hosts127.0.0.1 localhost::1 localhost ip6-localhost ip6-loopback172.17.0.10 nginx1

$ docker exec c1 printenv

(See for yourself)

Copyright Reserved

Access Control using icc and links

• The --link option allows access to the exposed ports (80, 443) of a source container (nginx1) from a target container (c1)

‒ Implemented using ACCEPT rules in iptables

24

$ sudo iptables –L -n...Chain FORWARD (policy ACCEPT)target prot opt source destinationACCEPT tcp -- 172.17.0.8 172.17.0.9 tcp spt:80ACCEPT tcp -- 172.17.0.9 172.17.0.8 tcp dpt:80ACCEPT tcp -- 172.17.0.8 172.17.0.9 tcp spt:443ACCEPT tcp -- 172.17.0.9 172.17.0.8 tcp dpt:443ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:443ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:80...

Insert by

link option

Exposed ports

in Dockerfile

Thank you.

https://github.com/sdnhub/lorispack© 2015 Copyright Reserved