Customizing Virtual Networks with Partial FPGA Reconfiguration
Dong Yin, Deepak Unnikrishnan, Yong Liao Lixin Gao and Russell Tessier
Funded by National Science Foundation Grant CNS-0831940
Electrical and Computer EngineeringUniversity of Massachusetts, Amherst
USA
2
Outline
Network virtualization
FPGA as a network virtualization platform
Customizing virtual routers with Partial FPGA reconfiguration
Dynamic virtual router allocation
Results
Conclusions
3
Network Virtualization
Many logical networks share a physical network infrastructure
Reduces costs Independent routing policies
Key Challenges
Isolation Performance Scalability Flexibility Usability
FA
C E
DB
C E
F
DB
A
B D
CE
A F
4
Network Virtualization Techniques
Software• Full/Container virtualization
ASIC/Network Processors• Supercharging PlanetLab • Juniper E series
S/W ASIC
Scalability High Limited
Flexibility High Limited
Performance Low High
Isolation Moderate High
Usability Good Good
5
FPGA as a Virtualization Platform
High performance Flexible hardware through reconfiguration
• Full reconfiguration• Reprogram entire chip• Chip shut down during reconfiguration
• Partial reconfiguration• Reprogram ‘part’ of the chip • Non-reconfigured regions can still operate during
reconfiguration
6
FPGA as a Virtualization Platform
Key Challenges
• Resource isolation between virtual routers • Scalability - Limited on-chip logic
In this paper,
• Use partial reconfiguration for better resource isolation between virtual routers
• Combine software virtual routers with hardware routers to create a scalable system
7
System Overview
SRAM
SRAM
PCI
1GEthernet
I/F
OpenVZVRouter
OpenVZVRouter
OpenVZVRouter
Kernel driverSoftware bridge
Linux
NetFPGA
PHY
SDRAM
SDRAMHW VRouter
HW VRouter
NIC NIC
8
Architecture
Fwd Logic
Fwd Table
Fwd Logic
Fwd Table
PRR1
PRR2
CPU TransceiverMAC RXQ
CPU RXQ
MAC RXQ
CPU RXQ
MAC RXQ
CPU RXQ
MAC RXQ
CPU RXQ
MAC TXQ
CPU TXQ
MAC TXQ
CPU TXQ
MAC TXQ
CPU TXQ
MAC TXQ
CPU TXQ
InputArbiter
DesignSelect
OutputQueues
PCII/F
Bridge
OpenVZ Virtual Router
OpenVZ Virtual Router
PCII/F
Bridge
VIP TYPEx.x.x.x HWy.y.y.y SW
VID01
CONTROL
Linux
FPGA Static
Reconfigurable Region
9
Fwd Logic
Fwd Table
Fwd Logic
Fwd Table
PRR1
PRR2
CPU TransceiverMAC RXQ
CPU RXQ
MAC RXQ
CPU RXQ
MAC RXQ
CPU RXQ
MAC RXQ
CPU RXQ
MAC TXQ
CPU TXQ
MAC TXQ
CPU TXQ
MAC TXQ
CPU TXQ
MAC TXQ
CPU TXQ
InputArbiter
DesignSelect
OutputQueues
PCII/F
Bridge
OpenVZ Virtual Router
OpenVZ Virtual Router
PCII/F
Bridge
VIP TYPEx.x.x.x HWy.y.y.y SW
VID01
CONTROL
FPGA
Linux
Architecture
Static
Reconfigurable Region
10
Dynamic Virtual Router Management
Virtual router requirements change over time
• Varying bandwidth/latency requirements
• Different routing policies -> Different h/w resources
Solution - Exploit heterogeneity
• High throughput routers -> Hardware• Low throughput routers -> Software
• Dynamically migrate virtual routers between s/w and h/w
11
Dynamic Virtual Router Management
Allocation• Try allocation in software (if b/w permits)• Else try allocation in hardware (if b/w and resources permit)
Removal• S/W - Destroy OpenVZ container • H/W - Program blank bitstream to reconfigurable region
Upgrades• Greedily migrate virtual routers between FPGA and Software
based on bandwidth requirement
12
Evaluation 2 Partially reconfigurable virtual routers on Virtex II
Software virtual routers - 3Ghz AMD X2 2GB RAM
Metrics
• Throughput• Reconfiguration time
Packet generation
• NetFPGA packet generator
SourceNetFPGA
Pktgen
VirtualRouter
SinkNetFPGA
Pktcap
1Gbps 1Gbps
13
Partial Bitstream Generation
Virtual router 1(Verilog)
Virtual router 1(Verilog)
Virtual router 2(Verilog)
Virtual router 2(Verilog)
Xilinx Early Access Partial Reconfiguration
PlanAhead
JTAG
Partial bitstream repository
14
Throughput of Partially Reconfigurable Virtual Router Consistent line rate (1Gbps) across packet sizes
15
Traffic Isolation and Reconfiguration Time
ethOpenVZ
OpenVZ eth
H/WVirtual Router A
H/W Virtual Router B
Sourc
e
Sin
k
MACQs
MACQs
DesignSelect
OutputQs
S/WBridge
S/WBridge
FPGA
Linux
Fully Reconfigure FPGA
Full Reconfiguration Approach
H/W Virtual router B’
16
Traffic Isolation and Reconfiguration time
20x reduction in
downtime
Full reconfiguration
Partial reconfiguration
12 seconds
17
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
1000000
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Number of virtual networks
Th
rou
gh
pu
t pe
r h
/w v
irtu
al r
ou
ter
(pp
s) )
Partial Reconfiguration
Full reconfiguration every 3 mins
Full reconfiguration every 30 sec
Throughput per virtual router Partial reconfiguration benefits at higher reconfiguration frequencies
18
Dynamic Virtual Network Allocation Evaluation Model
• 1000 virtual networks• Bandwidth distribution from PlanetLab nodes.• Poisson arrivals and Poisson lifetimes
• Mean arrival period = 2hrs• Mean lifetime = ~2.5 days (64 hrs)
• Bandwidth of live network changes according to Uniform distribution between 0%-X% from initial allocation b/w
• All networks can be either allocated in FPGA or Software virtual routers
19
Benefit of Virtual Network Migration Upto 20% more upgrade requests serviced by dynamic assignment
Reallocation benefits over wide fluctuations in bandwidth
20
Resource Usage and Power Consumption
Power management via Clock gating• Shutdown virtual routers when not in use• Saves 10% overall power
FPGA # Partially Reconfigurable Virtual Routers
Virtex II 2
Virtex 5 32
21
Conclusion A novel heterogeneous network virtualization platform
• Enhanced resource isolation and reconfiguration time• With partial FPGA reconfiguration
• Scalable • Combines fast FPGA virtual routers with software virtual
routers• Dynamic network allocation and migration
• Towards a green virtualization platform• Integrates power management techniques
22
Future Work
Usability• Higher level abstractions for creating and managing
partial bitstreams
Explore non-JTAG interfaces for reconfiguration (PCI Express/ICAP)
23
Thank you
Questions?
