PresenterVivekanandan Narasimhan
Distributed Virtual RoutersOpenstack India Meetup
Agenda
Introduction
High level architecture and DVR Configuration
East-West Routing
Distributed Virtual Routers Openstack India Meetup
Performance
North-South Routing
Services
Legacy Routing in NeutronNetwork node provides:
• IP forwarding
– Inter-subnet (east-west) traffic between VMs
– Floating IP (north-south) traffic between external and VM
– Default SNAT (north-south) traffic from VM to external
• Metadata Agent
– access to Nova metadata service
Issues:
• Performance bottleneck
• Scalability limitations
• Single Point of Failure
Compute node
Network node
Compute node
VM1 VM2
VM1 VM2
Virtual
Physical
Router A
RtrA
OpenStack India MeetupDistributed Virtual Routers
Distributed Routing in NeutronCompute node provides:
• IP forwarding for local VMs
– Inter-subnet (east-west) traffic between VMs
– Floating IP (north-south) traffic between external and VM
• Metadata Agent for local VMs
– access to Nova metadata service
Advantages:
• Bypass network node improves performance
• Scales with size of compute farm
• Limited failure domain (per compute node)
Limitations:
• Default SNAT function is still centralized
Compute node
Network node
Compute node
VM1 VM2
VM1 VM2
Virtual
Physical
Router A
RtrADefault
SNATRtrA’’RtrA’
Distributed Virtual Routers OpenStack® India Meetup
High level requirements for DVR
• Help close the parity gap with Nova (multi-host)
• Provider feature
– Tenants should not have to know or care
• Configurable on a per-router basis (centralized or distributed)
– Default router type set by a global config knob
• Can be deployed into existing environments
• Centralized routers and Distributed routers can coexist in same cloud
• Ability to migrate a router from Centralized to Distributed
• Minimize overhead use of public IP addresses
• Leverage existing code base
OpenStack® India MeetupDistributed Virtual Routers
DVR High Level Architecture and Configuration
Distributed Virtual Routers OpenStack India Meetup
Network/Service Node(s)
Compute Node(s)
VMVM
VM
l3-agentovs-agent
Controller Node
Neutron controller/plugin L3 Plugin
Network/Service Node(s)
l3-agentovs-agent
dhcp-agent qr
qrqr
agent_mode=dvr
router_distributed=True
enable_distributed_routing=Trueagent_mode=dvr_snat
enable_distributed_routing=True
• One FIP namespace supports multiple routers
• Legacy OpenStack Routers (non-distributed) can exist on the network/service node
• Enhanced l3-agent works in different modes “dvr_snat”, “legacy” and “dvr”.
FIP
SNAT
Distributed Virtual Routers
metadata
metadata
qrqr
DVR High level Architecture
OpenStack India Meetup
DVR - Features
The DVR VLAN effort in Kilo enabled the following:
DVR can route across two VLAN networks
DVR can route between a VLAN network and VXLAN network
DVR can route between a VLAN network and GRE network
Retained from Juno:
DVR continue to route between two VXLAN networks
DVR continue to route between two GRE networks
Distributed Virtual Routers OpenStack India Meetup
“neutron.conf”
router_distributed=True
Plugin Configuration ( Default Router Option)
“l3_agent.ini”
agent_mode=dvr_snat ( Network/Service Node)
agent_mode=dvr (Compute Node only)
agent_mode=legacy ( Network/Service Node)
router_delete_namespaces=True ( Enable namespace
cleanup)(Default)
L3 Agent Configuration and Modes
DVR Configuration Overview
Distributed Virtual Routers OpenStack India Meetup
“ovs_neutron_plugin.ini”
enable_distributed_routing = True
enable_tunneling = True
local_ip = <Local Data Network IP (or) TEP IP>
tunnel_types = vxlan
l2_population=True
L2/OVS Agent Configuration
DevStack Configuration
“local.conf”
Q_DVR_MODE=dvr_snat/dvr/legacy (Default)
DVR Configuration Overview
Distributed Virtual Routers OpenStack India Meetup
East-West Routing
Distributed Virtual Routers OpenStack India Meetup
L3 Agent
Rules------------
Rules------------
vm1 vm2
br-int
br-tun
qr
LMAC
East-West routing is accomplished through a combination of the following in each CN:
•qr : a namespace that forwards traffic among routed subnets using route table entries.
•LMAC: a MAC address unique to each CN for use as source address in forwarding routed frames from one CN to another.
•A set of rules in the OVS bridges that prevent frames with src=gateway MAC from egressed tunnel bridge or the IR. These rules also cause peer bridges to substitute/restore gateway MACs with LMACs while routed frames traverse
through br-tuns and underlay network.
br-ex
Compute Node entities for enabling the DVR East West Routing
L3 Agent
Distributed Virtual Routers OpenStack India Meetup
vm1
br-int-cn1
br-tun-cn1
qr
vm2
br-int-cn2
br-tun-cn2
qr
Tenant has two VMs - vm1 that is in RED Net & vm2 that is in GREEN NetCN1 and CN2 are two compute nodes. qr represents a dvr owned by Tenant 1
Data Network
CN1 CN2
PING REQUEST from vm1 to vm2
vm1 ipqr red macvm1 mac vm2 ip
vm1 ipqr red mac
vm2 ip
vm1 ipvm2 macqr grn mac vm2 ip
vm1 ipvm2 macl_mac vm2 ip
vm1 ipvm2 macl_mac vm2 ip
1
2
3
4
5
grn vni
grn vlan
vm1 ipvm2 macqr grn mac vm2 ip6
vm1 mac
Flow of DVR routed packets (example)
Distributed Virtual Routers OpenStack India Meetup
North-South Routing
Distributed Virtual Routers OpenStack India Meetup
Overview of North-South Routing
External Network
vm1 vm3
br-int
br-tun
vm5
br-ex
• IP Rules
•Routing Entries
• Iptable based NATing rules
• /32 addresses for the floating IPs
• One Floating IP Namespace per CN
• Interface on external network
• Proxy ARPs enabled for hosted
floating IPs
qr
FIP
L3 Agent
Data Network
Distributed Virtual Routers OpenStack India Meetup
Distributed Virtual Routers
Default SNAT Traffic flow
OpenStack India Meetup
Services
Distributed Virtual Routers OpenStack India Meetup
Services support with DVR
Services Support in DVR as of Kilo
LBaaS
FWaaS• North-South only ( External Traffic)
MetadataService
VPNaaS support for DVR .
Distributed Virtual Routers OpenStack India Meetup
Performance
Distributed Virtual Routers OpenStack India Meetup
• North/South Traffic for FIPs: VMs have direct access to external network, bypassing the “network node”
• East/West Traffic: VMs which exists in the same hypervisor see the most performance gain. VMs in different hypervisor see some gain also.
Performance benefits of DVR
Distributed Virtual Routers OpenStack India Meetup
Hypervisor
Hypervisor
Hypervisor
Hypervisor
vms
vms
vmsvms
NetworkNode
Centralized Virtual Router(CVR)
Physical Switch
Outside workd(ext-net
DVR East/West Traffic
DVR North/South FIP Traffic
CVR/legacy East/West Traffic
CVR/legacy North/South FIP Traffic
Traffic flow differences
Hypervisor
Hypervisor
Hypervisor
Hypervisor
vms
vms
vmsvms
NetworkNode
Centralized Virtual Router
(CVR/legacy)Physical Switch
Hypervisor
Hypervisor
Hypervisor
Hypervisor
vms
vms
vmsvms
NetworkNode
Distributed Virtual Routing(DVR)
Physical Switch
Hypervisor
Hypervisor
Hypervisor
Hypervisorvms
vms
vmsvms
NetworkNode
WITH DVR Enabled
Outside world(internet for example)
Outside world(internet for example)
External Gateway
External Gateway
Physical Switch
IR
External Gateway
Distributed Virtual Routers OpenStack India Meetup
Vms inDifferentSubnets
Vms in DifferentSubnets
NORTH/SOUTH
Simultaneous Individual
Mbps Mbps %of Max
dvr_vm_west 5272 9283 99%
dvr_vm_east 2123 8972 96%
cvr_vm_west 1168 1939 21%
cvr_vm_east 809 2083 22%
Total(Max) 9372
netperf -l 20 -H 192.0.2.10EAST/WEST
Simultaneous Flows
From To Mbps Aggregate
dvr_vm_west dvr_vm_east 4858
dvr_vm_east dvr_vm_west 5156 10014
cvr_vm_west cvr_vm_east 576
cvr_vm_east cvr_vm_west 431 1007
netperf -l 20 -H <to_vm>
Some quick test results
Distributed Virtual Routers OpenStack India Meetup
THANK YOU
Distributed Virtual Routers OpenStack India Meetup
Questions?
Distributed Virtual Routers OpenStack India Meetup