Build High Performance NSH-based SFC Solution with FD.io and OpenDaylight *Danny Zhou ([email protected]) , Yi Yang ([email protected])
Network Platforms Group, DCG, Intel
Acknowledgement: Hongjun Ni, Keith Burn, Brady Johnson, Anna Wan, John DiGiglio
2
Agenda
Service Function Chaining Overview
FD.io’s NSH_SFC Plugin
– Internals
– Evolving NSH_SFC Features
– NSH_SFC Performance and Analysis
– Functional and Performance Automation Test
OpenDaylight SFC Integration with VPP and NSH_SFC
Summary and Future Plan
3
Introducing SFC (Service Function Chaining)
SFF1
SF1(FW)
SFF2
SF2(LB)
Classifier1
Classifier2
Legend
SFC ComponentSFF: Service Function Forwarder
SF: Service Function
NSH: Network Service HeaderService Function Chain
Client1
Client2
Client3
Server
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
VER|O|C|R|R|R|R|R|R|R| Length | MD Type | Next Protocol
Service Path ID | Service Index
Mandatory Context Header
Mandatory Context Header
Mandatory Context Header
Mandatory Context Header
Proxy
SFC Use Cases Overview
4
Domain Use Cases Details
Telco Gi-LAN in Mobile Core Network
Internet Gateway in Fixed Broadband Network
Internet Gateway in Cloud CPE
Data Center
Internet Gateway
Customer
Network
Access
NodeBNG
Internet
Service Chain Network
L2-CPE
L2-CPE
Internet
Cloud CPE / SFC
FW
ACL
NAT
…
Server
Server
… Router Internet
Data Center
IDS/IPS FW NAT
SFC
UE SGW PGWInternet
SFC
ADCFW
DNS
PCEF
Video Opt.
DPI
FD.io’s NSH_SFC Plugin Internals
5L4 header
InnerIP header
Inner MACheader
UDP headerOuter
IP headerOuter MAC
headerNSH
headerVxLAN-GPE
headerPayload
• Typical NSH packet processed by VPP and NSH_SFC plugin
VxLAN-GPE
NSH_SFC Plugin
NSH Map NSH Entry
nsh-inputnsh-proxynsh-
classifier
nsh-aware-vnf-proxy
Supported Graph Nodes
nsh-inputnsh-proxynsh-
classifier
VxLAN-GPE
Maps one tunnel to another
Stores NSH header info
L2-input-classify
Ethernet-input
inputOtherGraph nodes
inputOtherGraph nodes
• NSH_SFC Plugin
Typical NSH_SFC Usage : NSH-Aware SF
VPPdpdk-input
ethernet-input
mpls-ethernet-input
ip6-input ip4-input arp-input llc-input
ip4-lookup
…
Packet Vector
NSH_SFC
VxLAN-GPE
FW
ip4-udp-lookup
ip4-local
NAT44
LB
ethernet-output
Legend
Ingress
Egress
ip4-rewrite-transmit
Evolving NSH_SFC Features
Service Function Forwarder Plugin framework in VPP and HC Integration with ODL SFC
16.09
NSH Classifier NSH Proxy Integration test with HC and ODL
17.01
NSH-aware SFs by collocating Proxy and SF (e.g. SNAT) Initial MD Type 2 support IOAM over NSH CSIT enabling (functional)
17.04
Under development ……. Enable Eth and Geneve as NSH transports Performance optimization Performance automation test Real NSH-aware VNF by enabling NSH_SFC to pass per-packet metadata to real SF (stretch goal)
17.07
NSH_SFC Performance and Analysis
9
• BIOS Configuration
Enhanced Intel Speedstep Enabled
Turbo Boost Enabled
Processor C3 Disabled
Processor C6 Disabled
Hyper-Threading Disabled
Intel VT-d Enabled
CPU Power and Performance Policy
Performance
Memory Freq. 2133 MHz
Total Memory Size 64 GB
Memory RAS and Performance Configuration -> NUMA Optimized
ENABLED
QPI B/W 9.6 GT/s
MLC Streamer ENABLED
MLC Spatial Prefetcher ENABLED
DCU Data Prefetcher ENABLED
DCU Instruction Prefetcher ENABLED
Direct Cache Access (DCA) ENABLED
• Software Configuration
OS Ubuntu 16.04 LTS
Kernel Linux version 4.4.0-21-generic
DPDK 16.11
VPP 17.01
NSH_SFC 17.01
Honeycomb 17.01
Device Under Test• Hardware Configuration
CPUIntel(R) Xeon(R) CPU E5-2699 v4 @
2.20GHz(Broadwell)
DIMM 2133 MHz, 64GB Total
NIC2x 82599ES 10-Gigabit SFI/SFP+
Network Connection
PacketGenIxia* 10 Gigabit Ethernet Traffic
Generator(16 ports)
DUT(VPP + NSH_SFC)
Traffic Gen(Trex)
Port A Port B
Port A Port B
Core 0
LegendFlow A
Flow B
• Network Topology
1C1T Throughput of NSH_SFC 17.01
72B 128B 256B 512B 1024B 1280B 1518B IMIX
NSH_SFC Classifier RFC2544 4934.449 7122.883 8257.485 8691.556 9777.756 9829.521 9841.643 8092.955
NSH_SFC Classifier 100% TX 5182.164 7174.001 9428.353 9669.746 9820.892 9854.23 9875.715 9425.316
Delta 5.02% 0.72% 14.18% 11.25% 0.44% 0.25% 0.35% 16.46%
Theoritical 10000 10000 10000 10000 10000 10000 10000 10000
0
2000
4000
6000
8000
10000
12000T
hro
ug
hp
ut
(Mb
ps)
NSH_SFC Classifier Throughput (RFC2544 vs. 100% TX Rate)
152B 256B 512B 1024B 1280B 1518B IMIX
NSH_SFC Proxy Inbound RFC2544 5401.172 8405.794 8269.983 9578.545 9661.54 9713.915 4566.067
NSH_SFC Proxy Inbound 100% TX 5401.565 8405.826 9172.955 9578.69 9661.58 9713.93 8633.353
Delta 0.01% 0.00% 10.92% 0.00% 0.00% 0.00% 89.08%
Theoritical 10000 10000 10000 10000 10000 10000 10000
0
2000
4000
6000
8000
10000
12000
Th
rou
gh
pu
t (M
bp
s)
NSH_SFC Proxy Inbound Throughput (RFC2544 vs. 100% TX Rate)
128B 256B 512B 1024B 1280B 1518B IMIX
NSH_SFC Proxy Outbound RFC2544 3843.331 8789.628 8729.212 9755.99 9819.183 9814.526 8700.796
NSH_SFC Proxy Outbound 100% TX 4807.208 8848.117 9640.68 9812.516 9848.719 9871.751 9422.431
Delta 25.08% 0.67% 10.44% 0.58% 0.30% 0.58% 8.29%
Theoritical 10000 10000 10000 10000 10000 10000 10000
0
2000
4000
6000
8000
10000
12000
Th
rou
gh
pu
t (M
bp
s)
NSH_SFC Proxy Outbound Throughput (RFC2544 vs. 100% TX Rate)
152B 256B 512B 1024B 1280B 1518B IMIX
NSH_SFC SFF RFC2544 4798.336 8623.209 8609.027 9739.464 9845.621 9869.394 8192.851
NSH_SFC SFF100% TX 5466.659 9205.381 9623.842 9808.213 9845.935 9869.751 9376.255
Delta 13.93% 6.75% 11.79% 0.71% 0.00% 0.00% 14.44%
Theoritical 10000 10000 10000 10000 10000 10000 10000
0
2000
4000
6000
8000
10000
12000
Th
rou
gh
pu
t (M
bp
s)
NSH_SFC SFF Throughput (RFC2544 vs. 100% TX Rate)
IMIX Profile for NSH_SFC
Packet Size(Bytes) 78 138 258 594 1518
Percentage 45.0% 23.5% 15.0% 11.0% 5.5%
NSH_SFC Performance Scaling and Performance Analysis
0
2
4
6
8
10
12
14
L2XC L2XC VxLAN IPFW VxLAN SFF NSH-aware SNAT
12.7
6.55.45
4.53.08T
hro
ug
hp
ut
(Mp
ps)
VPP and NSH_SFC Performance Comparision
(1C1T)
Source: VPP 17.01 CSIT Performance Report and NSH_SFC Performance Report
NSH ClassifierNSH Proxy
Inbound
NSH Proxy
Outound
NSH Proxy
BidirectionNSH SFF
1 core 5,232.27 5,404.78 4,785.75 4,835.17 5,504.20
2 cores 10,063.19 10,277.84 9,113.27 9,685.17 10,500.08
0.00
2,000.00
4,000.00
6,000.00
8,000.00
10,000.00
12,000.00
Th
rou
gh
pu
t (M
bp
s)
NSH_SFC Scaling Test Throughput
VPP & NSH_SFC Functionality Description
L2XC (64B) L2 Cross Connect
L2XC VxLAN (64B + 50B)
L2 Cross Connect + VxLAN Encap/Decap
SFF (128B)L2 Cross Connect + VxLAN-GPE
Encap/Decap + NSH manipulation
NSH-awareSNAT (128B)
L2 Cross Connect + VxLAN-GPE Encap/Decap + NSH manipulation + SNAT• 2 to 3 dedicated cores achieves 10G line
rate for smallest packet sizes
Test Case Packet size
NSH Classifier 72B
NSH Proxy Inbound 152B
NSH Proxy Outbound 128B
NSH SFF 152B
DUT2(VPP)
DUT1(VPP )
Traffic Gen
DUT(VPP + NSH_SFC)
Traffic Gen
End-to-End Throughput of NSH_SFC on Single Server
DUT
Niantic Port A Niantic Port B
VPP(L3 routing)
Vhost_user Port A
Container A
VPP(L2 xConnect)
Niantic PMD
Virtio_user
Traffic Gen
NianticPMD
DUT
Niantic Port A Niantic Port B
VPP(L3 routing)
Vhost_user Port A
Container A
VPP(L2 xConnect)
Niantic PMD
Virtio_user
Traffic Gen
Container B
VPP(L2 xConnect)
Niantic PMD
Virtio_user
Vhost_user Port B
Test NameVPP Functionality
(1C1T for each VPP instance)Packet Size (Byte) VPP-17.01
(Mpps)
CV L2XC + L3 routing 64 5.35
CVC L2XC + L3 routing + L2XC 64 3.59
CVC NSH-aware SNAT + SFF + NSH-aware SNAT 128 1.09
• CV setup • CVC setup
Function and Performance Automation Test
NSH_SFC Feature and Performance Test in CSIT Framework Example scripts to configure Service Function Forwarder on DUT1
/* Configure IP, routing and ARP tables */
set int state TenGigabitEthernet5/0/0 up
set int ip table TenGigabitEthernet5/0/0 0
set int ip address TenGigabitEthernet5/0/0 192.168.50.72/24
set int state TenGigabitEthernet5/0/1 up
set int ip table TenGigabitEthernet5/0/1 0
set int ip address TenGigabitEthernet5/0/1 192.168.50.73/24 ip route add 192.168.50.74/24 via 192.168.50.73
set ip arp TenGigabitEthernet5/0/1 192.168.50.74 02fe.8629.b438
/* Configure P2P VxLAN tunnels */
create vxlan-gpe tunnel local 192.168.50.72 remote 192.168.50.71 vni 9 next-nsh
encap-vrf-id 0 decap-vrf-id 0
create vxlan-gpe tunnel local 192.168.50.73 remote 192.168.50.74 vni 9 next-nsh
encap-vrf-id 0 decap-vrf-id 0
/* Configure NSH map and entry tables */
• create nsh entry nsp 185 nsi 255 md-type 1 c1 1 c2 2 c3 3 c4 4 next-ethernet
• create nsh entry nsp 185 nsi 254 md-type 1 c1 11 c2 12 c3 13 c4 14 next-ethernet
• create nsh map nsp 185 nsi 255 mapped-nsp 185 mapped-nsi 254 nsh_action swap
encap-vxlan-gpe-intf 4
Feature automation test enabled, performance automation test WIP
DUT1(Classifier/SFF/Proxy)
DUT2(L2Xconnect)
Traffic Gen(Trex)
192.168.50.73 192.168.50.74
VxLAN tunnel
192.168.50.71
192.168.50.72
OpenDaylight SFC Integration with VPP and NSH_SFC
16
Architecture: SFC VPP Renderer and Classifier in ODL SFC
SFC Provider
Data Store
SFC UI RESTCONF
OpenFlow Renderer VPP RendererSFC OvS*
Openflow PluginNetconfPlugin
OVSDB Plugin
Switch (OvS*)Netconf Device
(VPP node)Data Plane
Devices
ODL
SFC
SBI
Not used for VPP
Used for VPP
VPP Classifier
Legend
17
Architecture: OpenDaylight SFC and VPP/NSH_SFC Integration
VPP
OpenDaylight* SFC
Honeycomb
HC Core
VPP Renderer
Netconf/YANG
VPP Core
JVPP Core
Core Binary APIs
HC NSH Plugin
VxLAN-GPE port
configuration
NSH_SFC plugin
JVPP NSH Plugin
NSH Entry and Map Table
configuration
JVPP Registry
VPP ConnectionNSH Binary APIs
Data Broker
Config Data Tree Operational Data Tree
Translation Layer
JVPP APIs JVPP APIs
NSH-related
Contributions led by Intel
Legend
VPP Classifier
VxLAN-GPE
18
Summary and Future Plan
High performance NSH-based SFC solution within VPP and OpenDaylight
– NSH_SFC plugin to support Eth transport to eliminate per-hop encap/decap for SFC in NFV environment
– vHost_user performance needs optimization for East-West traffics
OpenDaylight SFC controls data planes mixed with VPP and DPDK OVS
Containerized SFC orchestrated by Kubernets and controlled by OpenDaylight
ONAP integration to provide orchestration support in SFC solution
– SDC supports SFC design
– MSO and Policy for SFC orchestration
– SDN-C and APP-C for data plane control
L4 packetinner
IP header
InnerMAC
header
NSH header,NP=0x3
Outer Ethernet,ET = 0x894F
Q&A
Summarized 1C1T Throughput of NSH_SFC 17.01
51205520
7088
9432
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
72 84 128 256
Th
rou
gh
pu
t (M
bp
s)
Packet Size (Bytes)
NSH Classifier Throughput for Different Packet Size
(1C1T)
7088
42504500
0
1000
2000
3000
4000
5000
6000
7000
8000
Th
rou
gh
pu
t (M
bp
s)
NSH Classifier NSH Proxy (inbound) SFF
NSH_SFC Throughput for 128B Packet
(1C1T)