Extensible Routers Using Network Processors

•Larry Peterson

•Scott Karlin

•Tammo Spalink

•Yitzchak Gottlieb

Princeton University April 19, 2023

Running Code On Routers

• Edge Routers– NAT, Firewalls, DiffServ, etc.

• Home Routers– Media gateway

• Scaleable servers– Level n switch

Princeton University April 19, 2023

Technology Forces

• Commodity switching fabrics– e.g., ATM switches, Infiniband SAN

• Programmable network processors– e.g., Intel IXP1200, Sitera PRISM

• Commodity processors– e.g., Intel IA32, IA64

Princeton University April 19, 2023

Research Problems

• Scheduling– SIGMETRICS 2001

• Programming– JSAC, March 2001

• Implementation– SOSP 2001

• Architecture– OpenARCH 2001

Princeton University April 19, 2023

VERA Architecture

. . . Network Services . . .

VirtualRouter

. . . Hardware Configurations . . .

Packet Flows

Forwarding Paths

Switching Paths

Princeton University April 19, 2023

Virtual Router

• Classifiers

• Schedulers

• Forwarders

Virtual Router

F

S

C

Princeton University April 19, 2023

Simple Example

F

Active Protocol

F

Proxy

F

IPSEC

F

IP

F

IP--

C S

Princeton University April 19, 2023

Smart Dropper

Layered Video Analysis(control plane)

(data plane)

Shared State

Control and Data Plane

Princeton University April 19, 2023

MicroEngines

Pentium

StrongArm

Processor Hierarchy

Princeton University April 19, 2023

Lab Setup

• IXP 1200– On Pentium III

Motherboard• Linux with driver

• PCI Bus

• Serial I/O

• 4 Pentium IIIs– Packet Sources/Sinks

– 2Kingston DEC 21143 ethernet cards

Princeton University April 19, 2023

Development Environment

• Linux– GCC/Binutil

• SA programs in C

– Commands and Libraries

• sgo, pcirestore

• Embedded C, LibCII, Runtime

• Windows– EEPROM Burner

– Assembler

VxWorks

Princeton University April 19, 2023

Scratch

DRAM

SRAM

6 Micro-Engines

StrongARM

FIFOs

IX B

us

MA

C P

orts

IXP1200 Chip

PC

I B

us

Intel IXP

Princeton University April 19, 2023

DRAM(buffers)

SRAM(queues,

state)

16 InputFIFO Slots

16 OutputFIFO Slots

16 InputContexts

8 OutputContexts

MicroEngines

Princeton University April 19, 2023

Evaluation

• Maximum forwarding rate– 3.47 Mpps (64-byte packets)

– Independent of ports and IXBus

RegisterOnly

DRAM32B

SRAM4B

Scratch4B

Input 171 2 3 6

Output 109 2 1 4

Total 280 4 4 10

Instruction Counts

Princeton University April 19, 2023

Main Loops

16 contextsinput loop:

until_rcvmove FIFO-to-DRAMlookup routeenqueue

8 contexts

output loop:dequeuemove DRAM-to-FIFOinit_xmit

Princeton University April 19, 2023

Main Loops

16 contextsinput loop:

until_rcvmove FIFO-to-DRAMlookup routenopnop...nopnopenqueue

8 contexts

output loop:dequeuemove DRAM-to-FIFOinit_xmit

Princeton University April 19, 2023

Virtual Router Processor

• Fixed Infrastructure (green code)– Can forward 1.13Mpps for 8 x 100Mbps ports

• Programmable VRP (red code) – Per 64-byte chunk

• All in registers

• 24 x 32-bit SRAM transfers (flow state)

• 240 register operations

• 3 hashes with hardware support

– Total of 650 instructions per MicroEngine

Princeton University April 19, 2023

StrongArm

• Handle exceptional packets

• Problems– Shares DRAM capacity with MicroEngines– On the critical path to Pentium

• Solution– Limit the role the SA can play

• Programmable bridge to Pentium

• µEngine manager

Princeton University April 19, 2023

Pentium

• Runs protocols in the control plane– e.g., BGP, OSPF, RSVP

• Run other router extensions– e.g., proxies, active protocols, overlays

• Scheduling is important– Packets can arrive too fast

Princeton University April 19, 2023

MicroEngines

Pentium

StrongArm

310 Kpps with1510 cycles/packet

3.47 Mpps w/ no VRP or1.13 Mpps w/ VRP budget

Performance

Linux

VERA

VRP

Princeton University April 19, 2023

Summary

• Extensible Routers are feasible– 240 Operations per 64-byte packet fragment– 24 SRAM Operations

• The processor hierarchy works– Use Line Cards (µEngines) for small tasks

• IP Forwarding

• Extensions

– Use Main Processor (Pentium) for large tasks

http://www.cs.princeton.edu/nsg

Implementation paper available July 2001