22
| Date post: | 29-Aug-2014 |
| Category: |
Technology |
| Upload: | janghoon-sim |
| View: | 2,723 times |
| Download: | 1 times |
● Prior Knowledge
● OpenStack Networking - VLAN
● OpenStack Networking - GRE
● Security Group, Floating-IP, NameSpace
● Neutron ML2
Index
Network Resources
Network Resources
Prior Knowledge - Network NameSpace
BMWNameSpace
eth0 eth1 eth2
Address
Routing table
Process Process
Process Process
Netfilter rules
eth0 eth1 eth2
BenzNameSpace
NetworkResources
NetworkResources
ProcessProcess
Process
Process
FordNameSpace
NetworkResources
Share
without Network NameSpace with Network NameSpace
Network NameSpace provides isolation of the system resources associated with networking. Thus, each network namespace has its own network devices, IP addresses, IP routing tables, /proc/net directory, port numbers, and so on. - http://lwn.net/Articles/531114/
Prior Knowledge - VLAN, GRE
VLAN - Virtual LAN
GRE - Generic Routing Encapsulation
16 Bytes Header + IP header Key field : 32bit
- identify an individual traffic flow within a tunnel
802.1Q Header TPIC : 16bit - 0x8100TCI : 16bit
PCP : 3bitDEI : 1bitVID : 12bit (0 ~ 4095)
OpenStack Installation - Grizzly
Controller node
Keystone
Network node Compute node - 1 Compute node - 2
Nova
Glance Horizon
Quantum L3-agentQuantum
openvswitch-agent
Nova compute
eth1 eth2
eth0
eth1 eth2
eth0
eth1 eth2
eth0
eth1 eth2
eth0
Management 192.168.20.0/24
Data 192.168.10.0/24
External network 192.168.122.0/24
Quantum openvswitch-agent
Quantum metadata-agent
Quantum dhcp-agent
Quantum openvswitch-agent
Nova compute
Quantum - Server
Network Topology
● ext_net : external network - 192.168.122.0/24● net_proj_one : “user_one” tenant - 50.50.1.0/24● net_proj_two : “user_one” tenant - 50.50.2.0/24● net_proj_new : “user_new” tenant - 60.60.1.0/24
Network node
net_proj_one net_proj_two net_proj_new
Big picture - VLAN
OpenStack Grizzly OpenvSwitch plug-in VLAN mode
Compute node - 1
br-ex
qg~
VM VM
br-eth1
tap~tag: 1
tap~tag:2
qg~ qg~
eth0
qr~
tap~ tap~ tap~
br-int
qr~ qr~
phy-br-eth1 Data 192.168.10.0
/24
OVS port
OVS Bridge
● qg~~~ : external gateway interface● qr~~~ : virtual router interface
int-br-eth1
eth1 eth1 br-eth1
phy-br-eth1
VM
tap~tag:2
br-intint-br-eth1
VLAN - Compute node
OpenStack Grizzly OpenvSwitch plug-in VLAN mode
Compute node - 1
VM VM
tap~tag: 1
tap~tag:2
br-
eth1
VM
tap~tag:2
Security Group[1]
Packet conversion
mod_vlan_vid
VM
tap~tag:3
br-intphy-br-eth1 int-br-eth1
eth1
veth pair
mod_vlan_vid
VLAN - Compute node
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-eth1NXST_FLOW reply (xid=0x4): cookie=0x0, duration=90455.716s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=2 actions=drop cookie=0x0, duration=89606.096s, table=0, n_packets=9484, n_bytes=2312018, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:1024,NORMAL cookie=0x0, duration=90456.248s, table=0, n_packets=6813, n_bytes=1325511, priority=1 actions=NORMAL
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-intNXST_FLOW reply (xid=0x4): cookie=0x0, duration=90458.482s, table=0, n_packets=64, n_bytes=4644, priority=2,in_port=1 actions=drop cookie=0x0, duration=89608.755s, table=0, n_packets=6499, n_bytes=1283680, priority=3,in_port=1,dl_vlan=1024 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=90459.075s, table=0, n_packets=9820, n_bytes=2323195, priority=1 actions=NORMAL
Packet conversion
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,idle_timeout=0,priority=3,in_port=1,dl_vlan=1024,actions=mod_vlan_vid:1,normal']Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-eth1', 'hard_timeout=0,idle_timeout=0,priority=4,in_port=2,dl_vlan=1,actions=mod_vlan_vid:1024,normal']
openvswitch-agent.log
NamespcaeNamespcaeNamespcae
VLAN - Network node
OpenStack Grizzly OpenvSwitch plug-in VLAN mode
eth0
qr~
tap~
qg~
qr~
qg~
qr~
qg~
br-int
br-ex
Packet conversion
mod_vlan_id
tap~ tap~
net_proj_one
net_proj_two
net_proj_new
Network node
Floating-IP(NAT)
eth1
br-eth1
int-br-eth1 phy-br-eth1
veth pair
mod_vlan_id
VLAN - Network node
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-intNXST_FLOW reply (xid=0x4): cookie=0x0, duration=7370.307s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=6 actions=drop cookie=0x0, duration=7368.424s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=2048 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=7367.991s, table=0, n_packets=764, n_bytes=191460, priority=3,in_port=6,dl_vlan=1024 actions=mod_vlan_vid:3,NORMAL cookie=0x0, duration=7369.073s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=500 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=7370.924s, table=0, n_packets=549, n_bytes=104066, priority=1 actions=NORMAL
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-eth1NXST_FLOW reply (xid=0x4): cookie=0x0, duration=7373.826s, table=0, n_packets=14, n_bytes=1104, priority=2,in_port=2 actions=drop cookie=0x0, duration=7372.725s, table=0, n_packets=13, n_bytes=922, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:500,NORMAL cookie=0x0, duration=7371.663s, table=0, n_packets=519, n_bytes=103966, priority=4,in_port=2,dl_vlan=3 actions=mod_vlan_vid:1024,NORMAL cookie=0x0, duration=7372.09s, table=0, n_packets=9, n_bytes=634, priority=4,in_port=2,dl_vlan=2 actions=mod_vlan_vid:2048,NORMAL cookie=0x0, duration=7374.384s, table=0, n_packets=764, n_bytes=191460, priority=1 actions=NORMAL
Packet conversion
Network node
net_proj_one net_proj_two net_proj_new
Big picture - GRE
OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
Compute node - 1
br-ex
qg~
VM VM
br-tun
tap~tag: 1
tap~tag:2
br-int
Tunnel
qg~ qg~
eth0
qr~
tap~ tap~ tap~
br-int
qr~ qr~
patch
patch b
r-tu
np
atch
gre~ g
re~
patch
Data 192.168.10.0
/24
OVS port
OVS Bridge
● qg~~~ : external gateway interface● qr~~~ : virtual router interface
Packet conversion
GRE - Compute node
OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
Compute node - 1
VM VM
tap~tag: 1
tap~tag:2
Tunnel
br-
tun
patch
gre
~
VM
tap~tag:2
Security Group[1]set_tunnel id
mod_vlan_vid
VM
tap~tag:3
br-intpatch
GRE - Compute node
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-tunNXST_FLOW reply (xid=0x4): cookie=0x0, duration=87770.027s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=87770.09s, table=0, n_packets=8786, n_bytes=1893724, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=87769.693s, table=0, n_packets=3031, n_bytes=617650, priority=3,tun_id=0x1,dl_dst=fa:16:3e:db:08:63 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=87769.966s, table=0, n_packets=6320, n_bytes=4432680, priority=3,tun_id=0x1,dl_dst=fa:16:3e:e0:73:95 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=87771.753s, table=0, n_packets=2921, n_bytes=951454, priority=1 actions=drop
Packet conversion
NamespcaeNamespcaeNamespcae
GRE - Network node
OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
br-tun
Tunnel
eth0
patch
gre~
qr~
tap~
qg~
qr~
qg~
qr~
qg~
br-int
br-ex
patch
Packet conversion
mod_vlan_id
set_tunnel id
tap~ tap~
net_proj_one
net_proj_two
net_proj_new
Network node
Floating-IP(NAT)
GRE - Network node
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-tunNXST_FLOW reply (xid=0x4): cookie=0x0, duration=474674.446s, table=0, n_packets=7899, n_bytes=2572502, priority=3,tun_id=0x3,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:2,output:1 cookie=0x0, duration=473163.123s, table=0, n_packets=7876, n_bytes=2565284, priority=3,tun_id=0x4,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:3,output:1 cookie=0x0, duration=633937.826s, table=0, n_packets=10543, n_bytes=3426814, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=473163.329s, table=0, n_packets=16484, n_bytes=3348666, priority=4,in_port=1,dl_vlan=3 actions=set_tunnel:0x4,NORMAL cookie=0x0, duration=474674.541s, table=0, n_packets=16864, n_bytes=3389132, priority=4,in_port=1,dl_vlan=2 actions=set_tunnel:0x3,NORMAL cookie=0x0, duration=633937.905s, table=0, n_packets=62044, n_bytes=37320316, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=472911.069s, table=0, n_packets=16335, n_bytes=3551350, priority=3,tun_id=0x4,dl_dst=fa:16:3e:89:fd:ce actions=mod_vlan_vid:3,NORMAL cookie=0x0, duration=474336.184s, table=0, n_packets=16360, n_bytes=3560332, priority=3,tun_id=0x3,dl_dst=fa:16:3e:d8:d5:29 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=474674.351s, table=0, n_packets=525, n_bytes=52427, priority=3,tun_id=0x3,dl_dst=fa:16:3e:69:ca:97 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=473162.912s, table=0, n_packets=197, n_bytes=19365, priority=3,tun_id=0x4,dl_dst=fa:16:3e:d6:b8:07 actions=mod_vlan_vid:3,NORMAL cookie=0x0, duration=633937.746s, table=0, n_packets=6207, n_bytes=630043, priority=3,tun_id=0x1,dl_dst=fa:16:3e:c7:ec:bd actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=474794.912s, table=0, n_packets=36912, n_bytes=7440964, priority=3,tun_id=0x1,dl_dst=fa:16:3e:8b:a6:d7 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=636252.069s, table=0, n_packets=163, n_bytes=36046, priority=1 actions=drop
Packet conversion
Security Group - VLAN, GRE
FORWARD
quantum-filter-top
quantum-openvswi-FORWARD
quantum-openvswi-local
quantum-openvswi-sg-chain
quantum-openvswi-iTAP_NUMBER
quantum-openvswi-oTAP_NUMBER
quantum-openvswi-sg-fallback
quantum-openvswi-sg-fallback
Security group is applied here
Security Group - VLAN, GRE
Chain quantum-openvswi-sg-chain (4 references)target prot opt source destination quantum-openvswi-i21767f1f-4 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap21767f1f-45 --physdev-is-bridgedquantum-openvswi-o21767f1f-4 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap21767f1f-45 --physdev-is-bridgedquantum-openvswi-i7903fd30-7 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap7903fd30-74 --physdev-is-bridgedquantum-openvswi-o7903fd30-7 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap7903fd30-74 --physdev-is-bridgedACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain quantum-openvswi-i7903fd30-7 (1 references)target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALIDRETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHEDRETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22RETURN udp -- 50.50.1.3 0.0.0.0/0 udp spt:67 dpt:68quantum-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain quantum-openvswi-o7903fd30-7 (2 references)target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 MAC ! FA:16:3E:DB:08:63RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67DROP all -- !50.50.1.2 0.0.0.0/0 DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALIDRETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHEDRETURN all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
[1] Note, OpenStack uses iptables rules on the TAP devices such as “tap~~” to implement security groups,.However, Open vSwitch is not compatible with iptables rules that are applied directly on TAP devices that are connected to an Open vSwitch port.
Network NameSpace
janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 ifconfiglo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 qg-fa243f49-d6 Link encap:Ethernet HWaddr fa:16:3e:9f:4b:63 inet addr:192.168.122.50 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe9f:4b63/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 qr-bc654dc2-f1 Link encap:Ethernet HWaddr fa:16:3e:c7:ec:bd inet addr:50.50.1.1 Bcast:50.50.1.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fec7:ecbd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 route Kernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Ifacedefault 192.168.122.1 0.0.0.0 UG 0 0 0 qg-fa243f49-d650.50.1.0 * 255.255.255.0 U 0 0 0 qr-bc654dc2-f1192.168.122.0 * 255.255.255.0 U 0 0 0 qg-fa243f49-d6
Floating-IP(NAT) - VLAN, GRE
janghoon@Network-node:~$ sudo ip netns showqdhcp-4c2f2346-ffaa-41a0-ab76-34cadf0163f5qrouter-e1b88ce4-51e9-4744-be80-d70d04c6a59bqdhcp-c19e22a0-1700-4b3b-91e5-2c961ef0a353qrouter-244fff3f-f935-4bdd-949d-739f1ce81dd0qdhcp-f37b681a-4be8-47b8-8063-3d17d24ee1aeqrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0
janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 iptables -L -n -t natChain quantum-l3-agent-PREROUTING (1 references)target prot opt source destination REDIRECT tcp -- 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697DNAT all -- 0.0.0.0/0 192.168.122.51 to:50.50.1.2
Chain quantum-l3-agent-float-snat (1 references)target prot opt source destination SNAT all -- 50.50.1.2 0.0.0.0/0 to:192.168.122.51
Chain quantum-l3-agent-snat (1 references)target prot opt source destination quantum-l3-agent-float-snat all -- 0.0.0.0/0 0.0.0.0/0 SNAT all -- 50.50.1.0/24 0.0.0.0/0 to:192.168.122.50
Floating-IP(NAT)
NameSpace
Neutron ML2
The Modular Layer 2 (ML2) plugin is a framework allowing OpenStack Networking to simultaneously utilize the variety of layer 2 networking technologies found in complex real-world data centers. It currently works with the existing openvswitch, linuxbridge, and hyperv L2 agents, and is intended to replace and deprecate the monolithic plugins associated with those L2 agents.
Neutron
TypeDriver
VLAN
ML2 Plugin
GRE VxLAN Flat
MechanismDriver
Op
envSwitch
Hyp
er-V
Op
enDaylig
ht
Arista
Cisco
Nexus
pSwitch
TypeDriver : TypeDrivers maintain any needed type-specific network state, and perform provider network validation and tenant network allocation. MechanismDriver : The MechanismDriver is responsible for taking the information established by the TypeDriver and ensuring that it is properly applied given the specific networking mechanisms that have been enabled.
https://wiki.openstack.org/wiki/Neutron/ML2
Neutron ML2
Network node Compute node - 1 Compute node - 2
Neutron L3-agentNeutron
ML2-agent
Nova compute
eth0
eth1 eth2 eth1 eth2
eth0
eth1 eth2
eth0
Neutron ML2 plugin
Neutron metadata-agent
Neutron dhcp-agent
Neutron ML2-agent
Nova compute
