TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc.
Software Datapath Acceleration for Stateless Packet Processing
June 22, 2010
Ravi MalhotraSoftware Architect
FTF-NET-F0817
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 2
► What can be accelerated• Stateless and stateful• Various applications• Sample TCP offload
►Soft Data Path Engine• Architecture• Feature set• Packet flow• DPE API
►Performance• Soft DPE advantage• Leverage key hardware offloads
Agenda
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc.
Stateful Path and Stateless Packet Processing
►Most network packet processing protocols can be broken down into two paths
• Stateless path, also known as the data path, requires quick and efficient switching/routing of packets
Can be broken down into packet identification (classification) and forwarding• Stateful path, also known as the control path, requires more processing and has
more inherent latency than the data path
►Stateful control path requires 90% of the code and is used 10% of the time.
►Stateless data path requires just 10% of the code and is used 90% of the time.
►This session focuses on how to accelerate the 10% of the code in the stateless path to increase packet processing performance.
3
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 4
Stateless Data Path for Different Applications
Application Data Path Control Path
Layer 2 bridging FDB lookup, VLAN add/delete, Learning
Aging, STP
IPv4 forwarding Dest-cache lookup, L2 modify
LPM route-table lookup, ARP, IP Options
NAPT 5-tuple lookup, IP/Port/L2 modify
Connection setup/destroy, policy, ALG
Firewall Access control list, pin-holes Stateful packet inspection, ALG
IPSec 5-tuple lookup, encap/decap+ crypto
SA setup, security policy
QoS Enforcement – sched, police, congestion, shaper
Policy, provisioning
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 5
Connection AgingSubsystem
Networking Stack
NetfilterConnection
Tracking
DPA Control Module Event Handlers
Asynchronous Low Level API
NetFilter Hooks Asynchronous Offload
Mechanism
Stateless Data Path Engine
Rule/Stream Tables
Data Path
Classifier/Action Table
Connection established/
assured event
New Connection
request
Create Rule
Create StreamAck Ack
Dynamic Connection Offloading with Soft DPA
(L4 TCP NAPT flow, no QoS)
Control Path
Lkup MissLkup HitPkt Flow
Ctrl Flow
Connection Offload Success
Probe Status
Run Aging
Ageoutlist
Aging
status
FIN/ACK (terminate pkt)
Connection destroyed
Delete Stream
Delete Rule
Conn destroy/ageout
Connection Deleted
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 6
Current – Linux Forwarding
eTSEC or FM-QM-BM eTSEC or FM-BM-QM
Linux Network Stack
eTSEC/QM Driver eTSEC/QM Driver
Control Plane Applications (DHCP/DNS/IGMP etc)
Linux KernelLinux User-space
e500 Cores
SEC (QM)SEC/QM Driver
Platform SoC
P1, P2 P1010, P1020, P2020, 85xx, 83xx
P3, P4, P5 P4080, P3040, P5020
Control Packet flow
Configuration flow
Data Packet flow
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 7
Stateless Data Path Procesing in QE
Network Interface Network Interface
Linux Network Stack
UCC UCC
Control Plane Applications (DHCP/DNS/IGMP etc)
Interworking microcode
Control LogicNetwork Driver
Linux Kernel
Linux User-space
RISC Microcode
DPE API
e500 Core
QE R
ISC C
ores
Existing Solution for 8360 and 8323
SEC
Completely re-used from
Linux
Platform SoC
P1, P2 8323, 8360, 8569
Control Packet flow
Configuration flow
Data Packet flow
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 8
Stateless Data Path Processing in Software
VeTSEC
Linux Network Stack
VeTSEC Driver VeTSEC Driver
Control Plane Apps (DHCP/DNS/IGMP/IKE etc)
Linux Kernel
Linux User-space
e500 Cores
Completely re-used from
QE based Platforms
SEC
VortiQa CP + NMS
Platform SoC
P1, P2 P1010, P1020, P2020, 85xx
P3, P4, P5 N/A
DPE API
SEC Driver
Control Packet flow
Configuration flow
Data Packet flow
VortiQa Network Stack
Soft Data Path Engine
VeTSEC
Control Logic
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 9
Soft Data Path Engine Feature List
►Stateless packet processing (all stateful processing including ALG, SPI firewall, ARP, routing, learning etc. done by control-path)
►Offloads following stateless processing• IPv4 forwarding• NAPT/firewall (ACL) processing• Layer 2 switching with VLAN• IPSec forwarding• Quality of service
►Support for the following interfaces:• Ethernet• VLAN• PPPoE• WLAN
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 10
Soft Data Path Engine Feature List (cont.)
►Maintenance• Per-flow statistics and aging
►Platform support• Multicore support over VeTSEC
►Provides a standard configuration across platforms• Integrates seamlessly with Linux networking stack and applications using
SWANG package• Integrates seamlessly with VortiQa networking stack and customer network
stacks
►Leverages hardware acceleration (hashing, scheduling, classification, security) where available
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 11
Data Path Engine
Soft Data Path Engine – Functional Model
Packet Parsing
Rx Processing
Classification/Lookup
Policer
Ingress HM-ops Egress HM ops
Scheduler/Shaper
Tx Processing
Backplane Processing / Inter-plane/processor communication
Control Plane (s)
Other DataPath Engine
Network Interface Network Interface
Recycle
Recycle
Application Offload– Crypto, PME etc.
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 12
ETHLogicalPort1
MA
C
Scheduler
Shaper
ETHLogicalPort2
MA
C
Scheduler
Shaper
Data Path Engine API Architecture Overview
LookupControl
Path
Tx Q1
Rx Queues
Stream1
BufferManager1
Stream2
BandwidthManager1
HdrMan1
Tx Q2
Stream3
Tx Q3 Send(data, stream3)
Match
Stream5
Rx Q1
Rx Q2
Stream4
error traffic
PHY Classification
Rule_1Rule_2Rule_3
BandwidthManager2
Tx queues
BufferManager2
HdrMan3
PHY HdrMan2
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc.
Soft Data Path Engine Performance Advantage
Results on P2020 RDB - 1200/600/400 : 2-core SMP Linux
13
Significant (2x to 5x) performance improvement over native Linux
IPv4 NAPT IPSec
0.00
1.00
2.00
3.00
4.00
5.00
6.00
0
500
1000
1500
2000
64 390 1500
% Increase
Thro
ughp
ut in
Kpp
s
NAPT ‐ Linux NAPT ‐ Soft DPA % Diff
0.00
0.50
1.00
1.50
2.00
2.50
0
500
1000
1500
2000
64 390 1500
% Increase
Thro
ughp
ut in
Kpp
s
IPv4 ‐ Linux IPv4 ‐ Soft DPA % Diff
0.00
0.50
1.00
1.50
2.00
2.50
3.00
0
100
200
300
400
500
600
64 390 1456
% Increase
Thro
ughp
ut in
Kpp
s
IPSec ‐ Linux IPSec ‐ Soft DPA % Diff
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc.
Soft Data Path Engine Multicore Scaling
Results on P2020 RDB - 1200/600/400: 1-core non-SMP vs. 2-core SMP Linux
14
Scaling factor of > 1.8x when migrating from 1-core to 2-core
Scaling Limited by SEC HW
IPv4 NAPT IPSec
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc.
Soft Data Path Engine Flow Scaling
Results on P2020 RDB - 1200/600/400 : 2-core SMP Linux – 64 byte traffic
15
Low performance degradation for handling multiple flows
IPv4 NAPT IPSec
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 16
Data Path Hardware Acceleration
Core(s)
Ingress Offload Egress OffloadAutonomous Processing
Look-AsideOffload
Drivers/API
Network Stack (SMP optimized)
GenericOffload
Autonomous aware
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 17
Hardware Acceleration Support
Offload Feature Advantage
Hash calculation Packet distribution to multiple cores, flow-pinning, table lookup
Coarse classification Offload stateless ACL processing
Ingress
Packet parsing Avoid software overhead
Hardware buffer management
No buffer alloc/free operations in softwareGeneric
Hardware queue management
Simpler packet Rx/Tx, efficient stashing (to L1/L2), leaves room in cache for other data
Egress Hardware QoS Avoid software overhead, mitigate DoS attacks, prioritize CPU cycles
Core Backside L2 cache Faster access for multiple flow tables
Look-Aside
Protocol-aware cryptography
Offload protocol encapsulation/decapsulation, sequence tracking etc.
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 18
Hardware Acceleration Advantage
100.00
150.00
200.00
250.00
300.00
Baseline IPv4 + QoS
Shaping + WFQ
WRED Policing Hash results in
FD
Parse results in
FD
HM ops in HW
HW Buffer Mgmt
HW Queue Mgmt
Stash on Dequeue
BS L2 cache
Relative speedu
p
Thro
ughp
ut in
kpp
s
Cycles Throughput (kpps) Absolute Tput %
Hardware Acceleration provides upto 2.5x improvement
TMFreescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, mobileGT, PowerQUICC, StarCore, and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMARTMOS, TurboLinkand VortiQa are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © 2010 Freescale Semiconductor, Inc. 19
Summary
► Software data path engine• Optimized packet processing path• Consistent interface across platforms• Easy integration with network stacks• Single solution across QorIQ LE/ULE platforms
► Performance advantage• Flexibility to leverage hardware acceleration• Optimized for multicore scaling
TM