EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 1 FABRIC 4 Gigabit Work &...

EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester1

FABRIC 4 Gigabit Work

&

VLBI-UDP Performance and Stability.

Richard Hughes-Jones The University of Manchester

www.hep.man.ac.uk/~rich/ then “Talks”

http://www.hep.man.ac.uk/~rich/


What is inside GÉANT2 What is the collaboration interesting?

10 Gigabit Ethernet UDP memory-2-memory flows TCP flows with allocated Bandwidth

Options using GÉANT Development Network 10 Gbit SDH Network

Options Using the GÉANT LightPath Service PoP Location for Network tests

Multi-Gigabit Trials on GEANT

Collaboration with Dante.


GÉANT2 Topology


GÉANT2: The Convergence Solution

NREN AccessNREN Access

ExistingIP Router

ExistingIP Router

GÉANT2POP B

GÉANT2POP A

Managed Lambda’s

1626 LM

1626 LM

L2Matrix

L2

TDM Matrix

TDM

1678 MCC

1678 MCCDar

k F

iber

EXPReS PC10 GE

EXPReS PC10 GE


What do we want to do? Set up 4 Gigabit Lightpath Between GÉANT PoPs

Collaboration with Dante PCs in their PoPs with 10 Gigabit NICs we FABRIC provide the SX XFPs, Dante the MCC ports

VLBI Tests: UDP Performance

Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems

Dante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP when using BW limited

Lightpaths

Need A Collaboration Agreement


Generic solution:

Rate adaptive mapping into VC4-nv with flow ctrl. and LCAS, n=1..64;

up to 4 ports per boardFull rate option:

Ethernet full rate up to VC4-68v, flow ctrl. and LCAS for n<68

Optical Transport Hierarchy option:

Mapping into OpticalDataUnit1/2 with flow ctrl.

O/EData

Mapper

4x 10GE LAN board in 1678MCC

DataMapper

MAC

MACO/E

O/E

BKPdrData

Mapper

DataMapper

MAC

MACO/E

Data Mapper Device - GFP-F into VC-4-nv - GFP-F into ODU1/2

4x10GE LAN interface ports

Tomatrix

10GE-LAN interface board in 1678 MCCDante interested in MCC performance


Options Using the GÉANT Development Network

10 Gigabit SDH backbone Alkatel 1678 MCC Node location:

London Amsterdam Paris Prague Frankfurt

Can do traffic routingso make long rtt paths

Available Feb 07 Less Pressure for

long term tests


Options Using the GÉANT LightPaths Set up 4 Gigabit Lightpath Between GÉANT PoPs

Collaboration with Dante PCs in Dante PoPs

10 Gigabit SDH backbone Alkatel 1678 MCC Node location:

Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna

Can do traffic routingso make long rtt paths

Ideal: London Copenhagen


Test PCs Have Arrived

Boston/Supermicro X7DBE Two Dual Core Intel Xeon

Woodcrest 5130 2 GHz Independent 1.33GHz FSBuses

530 MHz FD Memory (serial)

Chipsets: Intel 5000P MCH – PCIe & MemoryESB2 – PCI-X GE etc.

PCI 3 8 lane PCIe buses 3* 133 MHz PCI-X

2 Gigabit Ethernet SATA


Lab Tests 10 Gigabit Ethernet 10 Gigabit Test Lab set up in Manchester

Cisco 7600 Cross Campus λ <1ms Server quality PCs Neterion NICs 4 Myricom 10 Gbit NICs – delivery this week Chelsio being purchased

B2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing

Installed Fedora Core5 Kernels 2.6.17 & 2.6.18 (+web100 + packet drop) on the Intel dual-core PCs (Stephen Kershaw)


iBOB


Prototype iBOB with two sampler boards attached

FPGA based signal processing board from UC Berkeley

See Ralph’s Talk this afternoon


Bryan Anderson

Station

Board

RAM

iBOB10GE

VSI

10GE CX4

systembasedDisk

VSI or headstack

CX4 - fibre media converter

iBOB block diagram

10 Gigabit Ethernet now available UDP/IP module exists Use for Demonstration of FPGA driven

IP networking Link iBoB to PC NIC – basic tests Test over GÉANT 4 Gigbit flow Onsala - Jodrell


vlbi_udp

Multiple stream tests over the Academic network


vlbi_udp Flow Diagram

Ring buffer

Input thread

Ring buffer

Send thread

Disk Memory

Receive thread

Output thread

Disk Memory

From Simon Casey

UDP Data

TCP Control

Control thread Control thread

Vlbi_control

vlbi_send vlbi_recv


Jodrell BankUK

DwingelooDWDM link

MedicinaItaly Torun

Poland

e-VLBI at the GÉANT2 Launch Jun 2005


Geant2 Launch event, June 2005


UDP Performance: 3 Flows on GÉANT Throughput: 5 Hour run Jodrell: JIVE

2.0 GHz dual Xeon – 2.4 GHz dual Xeon670-840 Mbit/s

Medicina (Bologna): JIVE 800 MHz PIII – mark623 1.2 GHz PIII 330 Mbit/s limited by sending PC

Torun: JIVE 2.4 GHz dual Xeon – mark575 1.2 GHz PIII 245-325 Mbit/s limited by security policing (>400Mbit/s 20 Mbit/s) ?

Throughput: 50 min period Period is ~17 min

BW 14Jun05

0

200

400

600

800

1000

0 500 1000 1500 2000Time 10s steps

Rec

v w

ire r

ate

Mbi

t/s

JodrellMedicinaTorun

BW 14Jun05

0

200

400

600

800

1000

200 250 300 350 400 450 500Time 10s steps

Rec

v w

ire r

ate

Mbi

t/s

JodrellMedicinaTorun


UDP Performance: Flows on GÉANT Packet Loss & Re-ordering

Jodrell: 2.0 GHz Xeon Loss 0 – 12% Reordering significant

Torun: 2.4 GHz Xeon Loss 6 - 12% Reordering in-significant

Torun 14Jun04

0

1

2

3

4

5

0 500 1000 1500 2000Time 10s

num

re-

orde

red

020000

400006000080000100000

120000140000

num

lost

re-ordered

num_lost

jbgig1-jivegig1_14Jun05

0

500

1000

1500

2000

0 500 1000 1500 2000Time 10s

num

re-

orde

red

0

50000

100000

150000

num

lost

re-ordered

num_lost


Instability: 2 streams into JIVE, March 2006From Simon Casey


Dedicated

Gbit link

EVN-NREN

OnsalaSweden

Gbit link

Jodrell BankUK

DwingelooDWDM linkMedicina

Italy

Chalmers University

of Technolog

y, Gothenbur

g

TorunPoland

Gbit link

Metsähovi


Any Questions?


Backup Slides


From PoS to EthernetConnect. Communicate. Collaborate

•More Economical Architecture

•Highest Overall Network Availability

•Flexibility (VLAN management)

•Highest Network Performance (Latency)

Router

IP Links

1/10 Gigabit Ethernet

VC-4-nv Channels

L2Matrix

TDM Matrix

1678 MCCTransport Node

VLANs


10 Gigabit Ethernet: UDP Throughput

1500 byte MTU gives ~ 2 Gbit/s Used 16144 byte MTU max user length 16080 DataTAG Supermicro PCs Dual 2.2 GHz Xenon CPU FSB 400 MHz PCI-X mmrbc 512 bytes wire rate throughput of 2.9 Gbit/s

CERN OpenLab HP Itanium PCs Dual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz PCI-X mmrbc 4096 bytes wire rate of 5.7 Gbit/s

SLAC Dell PCs giving a Dual 3.0 GHz Xenon CPU FSB 533 MHz PCI-X mmrbc 4096 bytes wire rate of 5.4 Gbit/s

an-al 10GE Xsum 512kbuf MTU16114 27Oct03

0

1000

2000

3000

4000

5000

6000

0 5 10 15 20 25 30 35 40Spacing between frames us

Rec

v W

ire

rate

Mb

its/

s

16080 bytes 14000 bytes 12000 bytes 10000 bytes 9000 bytes 8000 bytes 7000 bytes 6000 bytes 5000 bytes 4000 bytes 3000 bytes 2000 bytes 1472 bytes


10 Gigabit Ethernet: Tuning PCI-X

16080 byte packets every 200 µs Intel PRO/10GbE LR Adapter PCI-X bus occupancy vs mmrbc

Measured times Times based on PCI-X times from

the logic analyser Expected throughput ~7 Gbit/s Measured 5.7 Gbit/s

mmrbc1024 bytes

mmrbc2048 bytes

mmrbc4096 bytes5.7Gbit/s

mmrbc512 bytes

CSR Access

PCI-X Sequence

Data Transfer

Interrupt & CSR UpdateKernel 2.6.1#17 HP Itanium Intel10GE Feb04

0

2

4

6

8

10

0 1000 2000 3000 4000 5000Max Memory Read Byte Count

PC

I-X

Tra

nsfe

r tim

e

us

measured Rate Gbit/srate from expected time Gbit/s Max throughput PCI-X

DataTAG Xeon 2.2 GHz

0

2

4

6

8

10

0 1000 2000 3000 4000 5000Max Memory Read Byte Count

PC

I-X

Tra

nsfe

r tim

e

us

measured Rate Gbit/srate from expected time Gbit/s Max throughput PCI-X

Date post:	12-Jan-2016
Category:	Documents
Upload:	adrian-robertson
View:	213 times
Download:	0 times

EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester 1 FABRIC 4 Gigabit Work &...

Documents