A Look at Intel’s Dataplane Development Kit

Dominik Scholz

Chair for Network Architectures and Services

Department for Computer ScienceTechnische Universitat Munchen

June 13, 2014

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 1

Outline

1 Packet Processing using Commodity Hardware

2 Intel’s Dataplane Development Kit

3 Comparison: Intel’s DPDK, netmap, PF RING DNA

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 2

Motivation

Why use commodity hardware and do packet processing insoftware?

Advantages:Flexibility: software can be modifiedIncreased performance and reduced costs of(multicore) CPU’s and NIC’s over the last yearsOpen source

But: existing dataplane software not designed for high-speedpacket processing (up to 10 Gbit/s)→ specialized frameworks implement different techniques toachieve significant performance speed-ups

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 3

Use Case: Linux Network Stack

Ingress Network Board Egress Network BoardMemory

Operating System

Buffer Buffer

1

9

Routing Table

2

3

4 5

78

10

11

Kernel Mode

User ModeApplikation

6

7

Applikation

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 4

Performance Limitating Factors

Bottleneck CPUBottleneck memory:

1 per packet allocation and deallocation2 multiple copy operations per packet3 complex sk buff structure

Parallelism: spinlocks (active waiting)Context switches

Conclusion: standard dataplane only for general purpose

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 5

Performance Limitating Factors

Bottleneck CPUBottleneck memory:

1 per packet allocation and deallocation2 multiple copy operations per packet3 complex sk buff structure

Parallelism: spinlocks (active waiting)Context switches

Conclusion: standard dataplane only for general purposeDominik Scholz: A Look at Intel’s Dataplane Development Kit 5

Performance Limitating Factors

Bottleneck CPUBottleneck memory:

1 per packet allocation and deallocation2 multiple copy operations per packet3 complex sk buff structure

Parallelism: spinlocks (active waiting)Context switches

Conclusion: standard dataplane only for general purpose

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 5

Outline

1 Packet Processing using Commodity Hardware

2 Intel’s Dataplane Development Kit

3 Comparison: Intel’s DPDK, netmap, PF RING DNA

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 6

Intel DPDK

Set of libraries to accelerate basic dataplane functionsReleased in 2012Completely replaces the network stackIntel architecture-based: supporting Intel Atom - Intel XeonOpen SourceBSD-licensed: free and unsupported standalone orcommercial solution

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 7

DPDK Overview

Runtime environment with lowoverhead

Dataplane libraries run inuserspace

1 Memory management2 Buffer management3 Custom driver4 ...

Environment AbstractionLayer (EAL)

”Easy to use.” - Intel

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 8

Queue Manager

Fixed-sized ring implemented as table of pointer to any object

Properties:FIFOLockless (no active waiting)Supports multi consumer/producer enqueue/dequeuescenariosSupports bunch-processing of objects

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 9

Memory Manager

mempool structure:Pool of fixed-sized objectsUses a ring to store free objectsPer core cache (optional)

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 10

Buffer Manager

mbuf structure used to store network packetsCreated before runtime”Allocation”: take a free mbuf from a mempool”Deallocation”: put the mbuf back to the mempoolSmall size to fit in one cache-line (→ mbuf-chaining)

mbuf contains:1 Metadata: control information, e.g. packet length2 Pointer to next mbuf3 Packet data: header and payload

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 11

How to use the DPDK - EAL

The DPDK creates libraries by creating the EAL:Hides environment specificsProvides standard programming interfaceOptimized for the available hardware

But does not provide:Layer-3 forwardingFirewalls...any layer 3 or upper protocol

→Developer has to port his application to the DPDK

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 12

Outline

1 Packet Processing using Commodity Hardware

2 Intel’s Dataplane Development Kit

3 Comparison: Intel’s DPDK, netmap, PF RING DNA

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 13

netmap

A framework for raw packet I/O, developed by Luigi Rizzo(Universita di Pisa)

Feature: works with broad range of soft- and hardware

Linux and FreeBSDIntel 10GbE and 1GbE adapterIntel, RealTek, nVidia

Implemented techniques:Memory pre-allocation and re-useMemory mappingBatch processingParallel direct paths (assign CPU core to receiving queue)

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 14

PF RING Direct NIC Access

A framework to capture packets, developed by ntop.

Feature: zero-copyPF RING DNA maps NIC memory and registers to userland→ only one copy operation per packetBut: weakness to user misbehaviour (system-crashes)

Implemented techniques:Memory pre-allocation and re-useMemory mapping (zero-copy)Parallel direct paths

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 15

Summary

Intel DPDK netmap PF RING DNAMemory Pre-allocation 3 3 3

Memory Mapping 3 3 3

Batch Processing 3 3 7

Parallel Direct Paths 3 3 3

Open Source 3 3 3

”Safety” 3 3 7

Test results show:different frameworks exceed in different use cases [2][4]up to 10 times faster than the linux network stack

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 16

Sources

Intel DPDKProgrammers Guide.January 2014.

Intel DPDKPacket Processing on Intel Architecture.Presentation slides, 2012.

Luigi Rizzonetmap: a novel framework for fast packet I/Oin: Proceedings of the 2012 USENIX Annual Technical Conference,2012.

Jose Luis Garcıa-Dorado et al.High-Performance Network Traffic Processing Systems UsingCommodity Hardwarein: Data Traffic Monitoring and Analysis, Springer Verlag, 2013.

www.dpdk.orgLast visited: 06.06.2014

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 17

Thank you for your attention!Do you have any questions?

Dominik Scholz: A Look at Intel’s Dataplane Development Kit 18