Agenda
• VPPOverview
• Structure
• Integrations
• Performance
VPPOverview
Introducing Vector PacketProcessor- VPP
Predictableperformance
NetworkIO
PacketProcessing:VPP
ManagementAgentNC/Y REST ...
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
Seealso:www.fd.io (TheFastDataProject)
Consistentperformance
Optimalperformance• 14+Mpps percore• Testedto480Gbps
• (bi-directional,24cores)• Multi-millionentryFIBs
• Directedgraphofnodes• Nodesareaconsistent,repeatablepattern
• Processesvectorofpackets
Introducing Vector PacketProcessor- VPP
NetworkIO
PacketProcessing:VPP
ManagementAgentNC/Y REST ...
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
Seealso:www.fd.io (TheFastDataProject)
Flexibilitywithoutsacrificingperformance
Introducing Vector PacketProcessor- VPP
• Multi-coreCPUsandleveragesDPDK
• RunsasaLinuxuser-spaceapplication
• Shipsaspartofbothembedded&serverproducts,involume
• Activedevelopmentsince2002
• RobustandcompleteAPIs• Approaching4th release• PackagedasDebian andRPMimagesfor:
• Ubuntu14.04,Ubuntu16.04,Centos7.2
16.06 16.091.0.0 17.01
Structurecodecomposition...plugins...API...whatmakesVPPtick...
VPPOverview
vppinfra• libraryofcomposable functionsfor
• vectors• memorymanagement• advanceddatastructures
• applicabletomanyproblemdomains• (seesimulated-annealing)
vnet
vlib
vppinfra
VPPOverview
vlib• VPPapplicationmanagement
• threading• main()• buffermetadatadefinition• CLI• ...more
vnet
vlib
vppinfra
VPPOverview
vnet• VPPpacketprocessingfeatures
API• AllfeatureshavefullAPIdefinedatthislayer
vnet
vlib
vppinfra
VPPOverview– vnet breakdown
• vnet codebrokenintofoldersperfeature
• mostfunctionalitycanbesummarizedbythesecategories
• eachdiscretefeatureimplementedasanodeinadirected-graph
Devices
Layer2
Layer3
Traffic Mgmt
Overlays
vnet
AF_PACKET
DPDK v16.11,HQOS
NETMAP
SSVM
QEMUvhost-user
multi-queue, indirectdescriptors
Ethernet,MPLSoverEthernet
HDLC,LLC,SNAP,PPP,SRP,LLDPVLAN- single/double tagging
MACLearningdefault limit of50kaddresses
BridgingSplit-horizon groupsupport/EFP FilteringVTR– push/pop/Translate (1:1,1:2,2:1,2:2)
Multimillion entryFIBs
SourceRPF
Thousands ofVRFsControlledcross-VRFlookups
Multipath– ECMPandUnequalCost
IPSec
IPv6
(Neighbor discovery,
RouterAdvertisement)
ARM32,
RaspberryPi
ProxyARP
ARPtermination
IRB– BVISupportwithRouter/MACassignment
Flooding
InputACLs
Interfacecross-connect
SPAN
BFD
TUN/TAPjumbosupport
CryptoDev
FIB2.0recursiveFIBlookup,failuredetectionIPMPLSFIBSharedFIBadjacencies
Mandatory Input Checks:TTLexpiration,Headerchecksum,L2length<IPlength,ARPresolution/snooping, perinterfacewhitelists
SNAT
MagLev-likeLoadBalancer
IdentifierLocatorAddressing(ILA)
Highperformanceportrangeingressfiltering
LawfulIntercept
Policer
GBP/SecurityGroupsclassifiersupport
Connection tracking
LISP
NSH
SegmentRouting
MAP/LW46
iOAM
DHCP
GREMPLS-GRENSH-GRE
VXLAN
VXLAN-GPE
L2TPv3
VXLANoverIPv6underlay
ARPproxy
SourceRPF
MultiplemillionClassifiers–ArbitraryN-tuple
Pluginsarefirstclasscitizens
• Pluginscanbein-tree:• SNAT,PolicyACL,L2Session,FlowPerPacket,ILA,IOAM,LB,SIXRD,VCGN
• Separatefd.io project:• NSH_SFC
ethernet-input
snat-out2in
vxlan-gpe-input
ip4-input
nsh-input
nsh-mapper
vxlan-gpe-input
APIcomposability
• Eachgraph-nodeandfeatureexposesabinaryAPI
• Accessibleviahigh-performanceshared-memoryring-buffer
• Implementedasasynchronouscallback
• Python*,Javabindings
vl_api_snat_add_address_range_t_handler()
vl_api_nsh_add_del_entry_t_handler()
vl_api_vxlan_gpe_add_del_tunnel_t_handler()
ExampleAPIcallfromuser
void add_l2_bridge (int bd_id, client_main_t *cm) { vl_api_bridge_domain_add_del_t *mp; ...mp->_vl_msg_id = ntohs (VL_API_BRIDGE_DOMAIN_ADD_DEL); mp->client_index = cm->my_client_index; mp->bd_id = ntohl(bd_id); ...mp->learn = 1; vl_msg_api_send_shmem (cm->vl_input_queue, (u8 *)&mp);
}
plugin
ethernet-input
snat-out2in
vxlan-gpe-input
ip4-input
nsh-input
nsh-mapper
vxlan-gpe-input
...butonlywhentheinstructionsareloadedin
here...
Howdoesitwork?
*approx.173nodesindefaultdeployment
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Nodeswilldoprocessing ...
1
2
...onthesethings ...3
...butonlywhenthedataisloadedinhere.
4
cpu instruction cache cpu datacache
Howdoesitwork? P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
cpu instruction cache cpu datacache
dispatch fn()
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
ethernet-input
Getpointertovector
PREFETCH#3and#4
PROCESS#1and#2
ASSUMEnext_node sameaslastpacket
Updatecounters,advancebuffers
Enqueue thepackettonext_node
<asabovebutsingleton>
whilepacketsinvector
while4ormorepackets
whileanypackets
Integrations
OPNFVFDS
16.09
https://wiki.opnfv.org/display/fds/FastDataStacks+Home
GBPRendererManager
GBPNeutronMapper
VPPrendererTopologyManager-
VBD
NeutronNorthBound
Neutron
...
Honeycomb(DataplaneAgent)
VPP
DPDK
System
Test(Yardstick)
System
Install(A
PEX)
Colorado
REST
NETCONFYANG
VPPML2
ML2MechanicalDriver
Neutron
...
VPP
DPDK
REST
17.01
ML2Agent
PythonAPI
REST
Containerintegration
CGO
CONTIV
VPP
DPDK
CAPI
17.01
GOLANG
containeragent
VPPdrivercontiv
FELIXv2
VPPdriver
PythonAPI
GOLANGAPI
ApplicationNetworking
Containernetworking
FIFO
PID1234
TCP
IP(routing)
device
send()
FIFO
PID4321
TCP
IP(routing)
device
recv()
kernel
glibc
Containernetworking
FIFO
PID1234
TCP
IP(routing)
device
send()
FIFO
PID4321
TCP
IP(routing)
device
recv()
FIFO
device
FIFO
device
VPP
af_packet
etc etcetcACL,SR,VXLAN-GPEip(4|6)-rewriteip(4|6)-inputethernet-input
dpdk
dpdk
device
af_packet
Whynotthis?PID1234 PID4321
recv()
VPP
FIFOFIFO
TCP
IP
DPDK
send()
DASHBOARD 60Gbpsround-trip
DASHBOARD40Gbpsoneway
VPPFIFO
vsLinuxbridge
iPerf
*reference:E5-2620v42.1GHz
Performance
VPP technology in a nutshell
• NDR = Non Drop Rate
• Read zero frame loss
• Not ITU 10^-7 packet loss ratio
• So simply put
• NDR here is better than ITU NDR J
OVSDPDK-vhost-user
VPP-vhost-user
0
1
2
3
4
5
6
7
NIC-to-VMVM-to-VM
OVSDPDK
VPP0
5
10
15
20
2MACs2kMACs
20kMACs
NDR rates for 2p10GE, 1 core, L2 NIC-to-NIC
[IMIX Gbps]
[IMIX Gbps]
NDR rates for 2p10GE, 1 core, L2 NIC-to-VM/VM-to-VM
• Virtual network infra benchmark of efficiency• All tests per connection only, single core• Higher performance with more connections, more cores• Latest SW: OVSDPDK 2.4.0, VPP 09/2015
OVSDPDK
VPP0.0
20.0
40.0
60.0
80.0
100.0
120.0
12routes 1kroutes100kroutes
500kroutes1Mroutes
2Mroutes
OVSDPDK
VPP0.0
10.0
20.0
30.0
40.0
50.0
60.0
6routes1kroutes
100kroutes500kroutes
1Mroutes2Mroutes
• VPPdataplanethroughputnotimpactedbylargeIPv4FIBsize
• OVSDPDKdataplanethroughputheavilyimpactedbyIPv4FIBsize
• VPPandOVSDPDKtestedonHaswell x86platformwithE5-2698v32x16C2.3GHz(Ubuntu14.04trusty)
NDRratesfor6port10GE,6cores,IPv4 NDRratesfor12port10GE,12cores,IPv4
[IMIX Gbps] [IMIX Gbps]
vNet-SLAbenchmarkingatscale:IPv4VPPandOVSDPDK
fd.ioFoundation27
UPDATEw/LATEST#s
Backupslides
...butonlywhentheinstructionsareloadedin
here...
Howdoesitwork?
*approx.173nodesindefaultdeployment
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Nodeswilldoprocessing ...
1
2
...onthesethings ...3
...butonlywhenthedataisloadedinhere.
4
cpu instructioncache cpu datacache
Howdoesitwork? P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
register node
init node
dispatch fn()
trace formatter
cpu instructioncache cpu datacache
Howdoesitwork? P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
cpu instruction cache cpu datacache
dispatch fn()
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
ethernet-input
Getpointertovector
PREFETCH#3and#4
PROCESS#1and#2
ASSUMEnext_node sameaslastpacket
Updatecounters,advancebuffers
Enqueue thepackettonext_node
<asabovebutsingleton>
whilepacketsinvector
while4ormorepackets
whileanypackets
Howdoesitwork?
cpu instruction cache cpu datacache
dispatch fn()
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
ethernet-input
PREFETCH#3and#4
Getpointertovector
PROCESS#1and#2
ASSUMEnext_node sameaslastpacket
Updatecounters,advancebuffers
Enqueue thepackettonext_node
<asabovebutsingleton>
whilepacketsinvector
while4ormorepackets
whileanypackets
1
2
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
P2 P3
Howdoesitwork?
cpu instruction cache cpu datacache
dispatch fn()
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
ethernet-input
Getpointertovector
PREFETCH#3and#4
PROCESS#1and#2
ASSUMEnext_node sameaslastpacket
Updatecounters,advancebuffers
Enqueue thepackettonext_node
<asabovebutsingleton>
whilepacketsinvector
while4ormorepackets
whileanypackets 2
1
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
P2 P3P0 P1
Howdoesitwork?
cpu instruction cache cpu datacache
dispatch fn()
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
ethernet-input
ASSUMEnext_node sameaslastpacket
Updatecounters,advancebuffers
Enqueue thepackettonext_node
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
P2 P3P0 P1
Getpointertovector
PREFETCH#3and#4
<asabovebutsingleton>
whilepacketsinvector
while4ormorepackets
whileanypackets
PROCESS#1and#2Checkandfix-upforany
packetsthataren’tgoing tothelast
packet’snext-node!
Howdoesitwork?
cpu datacache
dispatch fn()
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
2
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
P2 P3 P4 P5
cpu instruction cache
ethernet-input
Getpointertovector
ASSUMEnext_node sameaslastpacket
Updatecounters,advancebuffers
Enqueue thepackettonext_node
<asabovebutsingleton>
whilepacketsinvector
while4ormorepackets
whileanypackets
1
PREFETCH#3and#4
PROCESS#1and#2
Howdoesitwork?
cpu datacache
dispatch fn()
ethernet-input
dpdk-inputaf-packet-input
vhost-user-input
mpls-inputlldp-input
...-no-checksum
ip4-input ip6-inputarp-inputcdp-input l2-input
ip4-lookup ip4-lookup-mulitcast
ip4-rewrite-transit
ip4-load-balance
ip4-midchain
mpls-policy-encap
interface-output
P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
P2 P3 P4 P5
cpu instruction cache
ethernet-input
Getpointertovector
ASSUMEnext_node sameaslastpacket
Updatecounters,advancebuffers
Enqueue thepackettonext_node
<asabovebutsingleton>
whilepacketsinvector
while4ormorepackets
whileanypackets
1
PREFETCH#3and#4
PROCESS#1and#2
VPPHistory
16.06 16.091.0.0 17.01pre-OSS
Devices
Layer2
Layer3
Traffic Mgmt
Overlays
vnet
DASHBOARD
14Mpps
Participatingprojects
Code
-#+#
Contributors: xxx
Coverity warnings:xxx
Integrations
contiv
openstack ML2
HoneycombCSITNSH_SFCONE_LISP
14
17
10
41
8
• UnitTestinfrastructure• maketest
• IPFIXenhancements• PNDAintegration
• Outputfeaturesupport• post- lookup
• Pythonbindings forplugins• Featuresub-grapharcsupport• iOAM enhancements
VPPHistory
16.06 16.091.0.0 17.01pre-OSS
Devices
Layer2
Layer3
Traffic Mgmt
Overlays
vnet
DASHBOARD
14Mpps
Participatingprojects
Code
-#+#
Contributors: xxx
Coverity warnings:xxx
Integrations
HoneycombCSITNSH_SFCONE_LISP
7
16
8
30
8
• Dynamicallyorderedsubgraphs• Doxygen generateddocs.fd.io• Java/Python languagebindings
VPPHistory
16.06 16.091.0.0 17.01pre-OSS
Devices
Layer2
Layer3
Traffic Mgmt
Overlays
vnet
DASHBOARD
14Mpps
Participatingprojects
Code
-#+#
Contributors: xxx
Coverity warnings:xxx
Integrations
HoneycombCSITNSH_SFCONE_LISP
7
16
8
19
8
VPPHistory
16.06 16.091.0.0 17.01pre-OSS
Devices
Layer2
Layer3
Traffic Mgmt
Overlays
vnet
DASHBOARD
14Mpps/core
Feb2016
Participatingprojects
Code
-#+#
Contributors: xxx
Coverity warnings:xxx
HoneycombCSIT
5
16
7
15
6
2002