Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer

CloudStack Collaboration

Conference 2012

Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer

Dominic Curran

Citrix XenServer Software Engineer

Nov 2012

Slides available under CC BY-SA 3.0


Contents

2

● Linux Bridge

● What does Open vSwitch Provide ?

● Open vSwitch Project

● How Open vSwitch works

● How Openflow works


What's wrong with Linux Bridge ?

3

● Linux Bridge:

– Nothing !

– L2 switch. Fast & reliable

– Simple forwarding

Bridge Kernel ModuleBridge Kernel Module

VMVMVMVM

VMVM

packet

● Open vSwitch:

– Full L2-L4 matching capability

– Decision in userspace

– ACLs, QoS & Bonding

– Mobility of state

OVS Kernel ModuleOVS Kernel Module

VMVMVMVM

VMVM

packet

Userspace ModuleUserspace Module


What does Open vSwitch Provide ?

4

● Provide greater visibility into the XenServer networking layer

● Provide distributed fine grained networking configuration and control policies

● Allows XenServer to be part of switching fabric

● Makes Network Admins happy.

● A Little History:

– Research project from Standford University

– Bay Area startup – Nicira

– Partnered with Nicira 2009


Open vSwitch Project

5

● http://openvswitch.org

● Developed under GPL & Apache 2 License

● Mailing Lists

– Announcements: [email protected]

– User-level discussion: [email protected]

– Dev (code review, etc): [email protected]

– Mail Archives available● As of Linux 3.3 patches started to be accepted for Open

vSwitch.


Open vSwitch Git Tree

6

● Source repository:

$ git clone git://openvswitch.org/openvswitch

vlan -main t

(Open vS

witch 1.0)

branch-1 .1

branch-1 .2

branch-1 .3

branch-1 .4

bran

ch-1

.4.1

bran

ch-1

.4.2

bran

ch-1.4

.3LTSLTS

branch-1.5

branch-1 .6

branch-1 .7

branch-1 .8

XS

6 .0

XS

6 .1

master

XS

6 .0 .2


Open vSwitch OpenSource Contributors

7

$ git log | grep Author | cut -d “:” -f 2 | sort | uniq -c | sort | -nr

In no particular order:

● Redhat

● IBM

● NEC

● HP

● Nicira

● Fujitsu

● Google

● Broadcom

● Cisco

● Intel

● Citrix


Main Components

8

● Open vSwitch– Open source: www.openvswitch.org

– Kernel mode module for packet switching (openvswitch_mod.ko)

– User space daemon for rules evaluation (ovsdb-server & ovs-switchd)

– Configured via OpenFlow protocol: www.openflow.org

● OpenFlow Controller– Open vSwitch has a number of nice standalone features

– BUT remote management is probably its most powerful

– Nearly every aspect can be controlled remotely

– Typically a deployment uses a centralized Controller

– DVSC (Distributed Virtual Switch Controller)

● One DVS Controller can manage multiple XenServer pools● Can be hosted in the same pool as it is managing● Provides Flexible way to group hosts for network management

XenServerHost

XenServerHost

OpenFlowController

OpenFlowController

vSwitchvSwitch

XenServerHost

XenServerHost

vSwitchvSwitch

XenServerHost

XenServerHost

vSwitchvSwitch


Architecture of Open vSwitch

9

● Each host runs a vSwitch userspace daemon plus kernel module

– ovs-vswitchd & ovsdb-server– Drop-in replacement for the Linux bridge based stack.– Daemon evaluates flow rules to determine destination of each frame.– Pushes rules down into the kernel as necessary.

● In isolation the vSwitch acts much as the Linux bridge does:

– Layer-2 MAC learning on physical and virtual interfaces.– From the XenServers toolstack level there is very little observable difference.– Under the hood the management tools, configuration of the stack etc are very

different.

● Optionally may connect to central Controller for distributed coordination and control


Main Components on one XenServer Host

10

ovsdb-server ovs-vswitchd

openvswitch_mod.koNetlink

Management Protocol (Unix Socket)

User

Kernel

JSON/RPC

FlowTableFlowTableFlowTable

Hash lookupTable

PacketClassifier


ovsdb-server & ovs-vswitchd

11

● ovsdb-server

– Database that holds switch-level configuration

– Custom database with properties of:● Value contraints● Weak references● Garbage collection

– Log-based (helpful from debugging)

– Speaks Management protocol to ovs-vswitchd & Controller

● ovs-vswitchd

– Core component in the system

– Supports multiple independent datapaths (bridges)

– Communicates● With controller using OpenFlow● With ovsdb-server using management protocol● With kernel module using netlink

● Packet classifier supports flow lookup with wildcard rules


openvswitch_mod.ko

12

● Fast path

● Kernel module handles packet switching

● Exact-match cache of flows

– Lookup & forwarding

● Designed to be fast and simple

– Knows nothing of Openflow

● Implements Tunnel encapsulation/decapsulation


How the Controller fits with the other Components

13


openvswitch_mod.koNetlink

Management Protocol (6632/TCP)

User

Kernel

JSON/RPC

Controller

OpenFlow Channel (6633/TCP)

Off-boxssl

xenbr1xenbr1 xenbr2xenbr2

FlowTableFlowTable

# ovs-vsctl …# ovs-appctl ...# ovs-ofctl ...

Command Line (ovs-vsctl, ovs-appctl, ..)

Hash lookupTable

PacketClassifier


XenServer Packet Flow – Packet Miss14


openvswitch_mod.ko

User

Kernel

JSON/RPC

DVS Controller

DomU ssl

xenbr1xenbr1 xenbr2xenbr2

FlowTableFlowTable

Hash lookupTable

eth1

Incoming Packet

Accessible via Browser

XAPIToolstack

Dom0

MISS #1

MISS #2PacketClassifier

OFPT_PACKET_IN


Flow Table Entries15 ● VM Charlie sends a ping to Bob.

# ovs-dpctl show xenbr0system@xapi0: lookups: hit:103033 missed:77944 lost:0 flows: 30 port 0: xenbr0 (internal) port 1: eth0 port 2: vif2.0

● Then dump flows:# ovs-dpctl dump-flows xenbr0 | grep "10.80.2.144"

in_port(2),eth(src=72:41:36:a2:8c:d9,dst=00:21:1b:f3:63:45),eth_type(0x0800),ipv4(src=10.80.228.31,dst=10.80.2.144,proto=1,tos=0,ttl=64,frag=no),icmp(type=8,code=0), packets:5013, bytes:491274, used:0.760s, actions:1

in_port(1),eth(src=00:21:1b:f3:63:45,dst=72:41:36:a2:8c:d9),eth_type(0x0800),ipv4(src=10.80.2.144,dst=10.80.228.31,proto=1,tos=0,ttl=62,frag=no),icmp(type=0,code=0), packets:5013, bytes:491274, used:0.760s, actions:2

● L2-L4 Exact Match

● Total Number of packet matches of this type● Total Number of bytes for this flow match● Time flow was last updated● Actions. In this case switch packet to Port 2.

xenbr0xenbr0

PIFeth0 vif2.0

VM- Domain210.80.228.31

(Charlie)

10.80.2.144(Bob)

Port 1 Port 2

Flo

w 1

Flo

w 2

XenServer Host

mailto:system@xapi0


What about SYN attacks ?

16

● SYN attack form of Denial-of-Service attacks

SYN

SYN-ACK

ACK

Client Server SYN

SYN-ACK

LegitimateClient

Server

OK

Attacker

SYN

TCP – Three Way Handshake

??● For Open vSwitch this is particular problem !

– Each SYN is a new flow, thus a user-kernel interaction

● Recent Open vSwitch introduced a 'Flow Governor'

– Detects large quantities of short-lived flows

– Stops flow setup in kernel module, thus missing a user to kernel interaction

Wait.. Wait..And wait...And wait...

Timeout!


Open vSwitch Features

17

● Supports OpenFlow 1.0

● Bonding

● 802.1Q VLAN

● Tunneling – Supports GRE mode

● Ingress policing

● NetFlow v5

● Port mirroring (SPAN and RSPAN)

● ACLs (through OpenFlow)

● Bridge compatibility

● Documented in ovs-vswitchd.conf man page:

– http://openvswitch.org/cgi-bin/ovsman.cgi?page=vswitchd%2Fovs-vswitchd.conf.5


What is OpenFlow ?

18

● OpenFlow allows the path of network packets to be setup by software.

● It is the enabler of 'Software Defined Networking'.

● Standard way to control flow-tables in commercial switches and routers.

● Gives Controller high level access to the datapath routing decisions of a switch.

● Currently, support is limited to 1.0 plus a number of extensions

● Now managed by Open Networking Foundation (ONF)


How does the Protocol work ?

19

● It uses a binary protocol for faster processing

● Header:

● Simple protocol for running on embedded CPUs in hardware switches

● vSwitch connects over SSL to port 6633 on the Controller.

version Type Length

8-bit 8-bit 16-bit

Transaction id

32-bit


OpenFlow Initial Setup Protocol

20

OFPT_HELLO

OFPT_HELLO

OFPT_ECHO_REQUEST

OFPT_ECHO_REPLY

OFPT_FEATURES

vSwitch Controller

OFPT_FEATURES_REPLY

Connection Established

- Datapath id- Capabilities- Actions- Ports - Name - HW Address - Port number

Message Types

● Controller-to-Switch

● Asynchronous

● Symmetric

- Version of vSwitch

- Version of Controller


OpenFlow Protocol21

OFPFC_ADD

OFPT_PACKET_IN

OFPT_STATS_REQUEST

vSwitch Controller

OFPT_STATS_REPLY

REPLY with:- Switch description- Flow statistics- Table statistics- Queue statistics- Port statistics

Reason to send: - Packet missOR - Explicit rule

- In Port- Reason - Buffer id- Data

OFPT_BARRIER_REQUEST

OFPT_BARRIER_REPLY

OFPFC_MODIFY

OFPFC_DELETE

Synchronize

Change entry in FlowTable


What happens if Controller connections is lost ?

22

● Immediately enter one of two modes:

– Fail secure mode

– Fail standalone mode

OFPT_ECHO_REQUESTvSwitch Controller?

ECHO_REQUEST timeout ???


WireShark Plugin for OpenFlow Protocol

23

● http://www.openflow.org/wk/index.php/OpenFlow_Wireshark_Dissector

● Command Line Alternative:

# ovs-ofctl snoop <bridge>

http://www.openflow.org/wk/index.php/OpenFlow_Wireshark_Dissector

Questions ?

24

● My Email: [email protected]

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

http://creativecommons.org/licenses/by-sa/3.0/deed.en_GB

Date post:	10-May-2015
Category:	Documents
Upload:	the-linux-foundation
View:	27,645 times
Download:	3 times

Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer

Documents