Date post: | 26-Dec-2015 |
Category: |
Documents |
Upload: | katrina-harris |
View: | 213 times |
Download: | 0 times |
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Networking for the Particle Physics Community - a UK Perspective
Introduction to the PPNCG Network TopologiesMonitoring ActivitiesPeering to the US and CanadaGrid Work SuperJANET4
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Introduction to the PPNCGMembership includes HEP and Astronomy users
–Ensure the community has the required networking facilities–Monitor end-to-end performance–Investigate new network applications / technologies–Provide advice on kit / facilities– URL icfamon.rl.ac.uk/ppncg/title.html
Acts as an Interface between HEP and UKERNA–Collaborations: Multicast, H.323, QoS tests, SuperJANET4
Active Network Monitoring–Uses the ICFA monitoring tools and the PPNCG traceping and ftp –Monitor MANs / SuperJANET & the networks to CERN, DESY, US–Report problems to UKERNA
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET Connections in the UK
Bishop Grosseteste
Warwick
Lancaster
Trinity
Afan
Merthyr
CoventryTechnical College
Northern College
Leeds
Bristol
Londo
n
Manch
ester CWC
O-BEN
R-BEN
Nott-BEN
Le-BEN
Ncl-BEN
Ed-BEN
C-BEN
ULCC-BEN
Cf-BEN
Be-BEN
Gla-BEN
Ex-BEN
Bham-BEN
Bris-BEN
155 Mbps
34 Mbps
CWC
Soton-BEN
Rdg-BEN
BWEMAN
MidMAN
LMAN
EMMAN
NorMAN
YHMAN
SWAN
SWMAN
FaTMAN
AbMANClydeNET
HomertonHinxton Hall
ITE Monkswood
BAS
Anglia Poly
Cambridge
RAL
Brookes
Oxford
Portsmouth
BEN
PoP Site
MAP
Birkbeck
UCL
IOE
SOASRVC LSHTM
Southbank
LondonInstitute
LondonGuildhall
Imperial
Westminster
KingstonRoehampton
LBS
RCA
Birmingham
Westhill
Newman
Coventry
UCEBWolverhampton
AstonQMW UEL
Telehouse
Goldsmiths
ULCC
Swansea
Carmarthen
Lampeter
Soton
Nottingham
Leeds
GreenwichRavensbourne
St Andrews College
Glasgow
Exeter
Plymouth
Dundee
Cardiff
Bristol
Belfast
St MaryStranmill
BGS
Bath
Bath College
HEFCE
UWE
RobertGordon
AberdeenSt Andrews
Abertay
ULCC
NewportWCM
DSwansea Institute
Glamorgan
St Davids
Cardiff Institute
GlasgowCaledonia Strathclyde
RSAMD
GSA
Paisley
EaStMAN
Edinburgh
Napier Queen Margaret
Stirling Heriot
Watt
ECA
Sunderland
Northumbria
DurhamUNN
Teesside
Nott Trent
Derby
HuddersfieldSheffield
Linc & Humb
Sheffield Hallam
Leeds Metropolitan
Hull
Scarborough
Bradford
York
MCC-BEN
MCC
NEWI
Bangor
G-MING
Salford
Manchester Metropolitan
RNCM
UMIST
Liverpool LJMU
NNW
Central Lancaster
Daresbury
PO Lab
ITE Bangor
Keele
Bolton Inst.
Liverpool Hope
Staffordshire
Edge Hill
Sussex
Brighton
Edinburgh
UHI
Chichester
Yale
ColegLlysfasi
Deeside
WCoH
Swansea College
Bridgend
Gorseinon
Aberdare
Neath Pembroke
Llandrillo Menai
Aberystwyth
HEAnet Ulster
HarlechPowys
Meirion-DwyforCeredigion
MST Radar
Falmouth
Darlington
BGSExeter
St Mark & John
Bucks
UKERNA
C&G
Seeseparate diagram
See separate diagram
Bournemouth
Royal HollowaySurreyReading
Cranfield
UEA
Essex
U-Net
Leicester
UC Northampton
UCL
CWC
Mid KentCollege
Stephenson
ChichesterCollege
Soton City
BCFTCS
LCC
City CollegeManchester
Halton
NewcastleNewcastle
CarlisleCollege
Cleveland College
New College Durham
Southport
SandwellWorcester Coll. CWC
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET – ULCC 3 * 155 Mbit links + 2 * 34 Mbit
Plus 3 * 155 Mbit links4 * 155 Mbit transatlantic links
Oct 00
SuperJANET Connections in London
August 2000
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
UK to Europe TEN-155 October 2000
Peering with CANARIE made in LondonUses 10 (15) Mbit PVC on the Dante-US line for all Europe4 * 155 Mbit European links operational
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Monitoring: ping plots UK - CANARIE Ping average round trip time & Lost packet % for icfamon.dl.ac.uk – cepheid.physics.utoronto.ca
15 Oct 00 100 byte packets 15 Oct 00 1000 byte packets
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Monitoring: US Traffic - Trends UKERNA Traffic Kbit/s. Blue Traffic from US Maroon Traffic to US10 Nov 1999 1 Nov 2000
UKERNA – US Traffic Daily totals Mbytes
Weekend
Summervacation
Weekday
John Macallister PPNCG OxfordXmas 99
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Monitoring: ping plots UK - US Ping average round trip time & Lost packet % for 100 byte packets icfamon.dl.ac.uk – ns2.slac.stanford.edu
13 Oct 99 15 Oct 00
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Peering with ESnetUsed CAR and WRED to give priority to Esnet traffic into JANET.Rtt and packet loss for 100 byte packets.Tests DL to SLACStart 21 Feb 00
CAR+WREDenabled
Config changes toCAR and WRED
Further configchanges
High Priority
Normal Priority
Robin Tasker PPNCG DL
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Peering with ESnet and Abilene rtt and packet loss for 100 byte packetsPlot start 16 May 00
ESnetHigh PriorityDL - SLAC
AbileneHigh PriorityDL - Stanford
Further configchanges
Change in bandwidthrestrictions from Abileneinto JANET
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
The Grid
Government has discussed 165M pounds to develop Computing Grid Infrastructure.Across Research Councils Initiative – not just Particle Physics Setting up management structure to co-ordinate PPARC activities.Have run Globus workshop so US developers could inform the community.Several sites now connected using Globus toolkit.Web site www.datagrid.org.uk
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
The Grid
Strong involvement in the EU DataGrid middleware proposal.
–Proposal accepted – Budget of 9.8M Euros
HEP experiments will need world Grid access.Grid Networking interests include:
–High throughput, reliable, very large data transfers–QoS issues – diffserv, RSVP and packet marking–Monitoring, Metrics, and prediction accuracy–Modelling
Physics Applications
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
The Grid: PAW with Globus
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDPLocal Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Toolkit
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDPLocal Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Toolkit
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDPLocal Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Toolkit
Linux GASS Server
Sun GASS Server
SGI GASS Server
CampusLAN
Zebra
HBOOK
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Core ServicesMetacomputing
Directory Service
GRAMGlobus
Security Interface
Heartbeat Monitor
Nexus
Gloperf
High-level Services and Tools
DUROC globusrunMPI Nimrod/GMPI-IO CC++
GlobusView Testbed Status
GASS
Local Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDPLocal Services
LSF
Condor MPI
NQEEasy
TCP
SolarisLinuxSGI
UDP
Toolkit
PAW
Next Steps:•Working on “root”•Brokers•Data Location•Security + Global farms
BaBar Tau Analysis
Andrew McNab Manchester
Demonstrated at theEU IST 2000 NiceData from Italy & UK
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
ATM MAN 7 routers ~0.8ms / router Router+link 0.4 s/byte DL-Liverpool ~10 ms rtt - agreement
The Grid:Measurements on SuperJANET
64 bytes pc-SJ-DL req-resp
0
100
200
300
400
500
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
256 bytes pc-SJ-DL req-resp
0
100
200
300
400
500
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
1024 bytes pc-SJ-DL req-resp
0
100
200
300
400
500
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)1500 bytes pc-SJ-DL req-resp
0
100
200
300
400
500
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
512 bytes pc-SJ-DL req-resp
0
100
200
300
400
500
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
Sigma ~1.2 ms
y = 2.2589x + 5643.6
y = 2.1519x + 5533.9
y = 0.107x + 109.71
0
2000
4000
6000
8000
10000
12000
0 500 1000 1500 2000 2500 3000
Message length bytes
La
ten
cy u
s
0
50
100
150
200
250
300
350
400
450
500
La
ten
cy f
or
PC
=P
C u
s
PC-SJIII-DL PCdt PC-SJIII-DL PCPC=PC
Latency Mancester - DL
Richard Hughes-Jones PPNCG Manchester
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
The Grid: Latency on SuperJANET
64 bytes pc-SJIII-sun req-resp
0
100
200
300
400
500
600
10000 15000 20000 25000 30000 35000 40000 45000
Latency us
N(t
)
256 bytes pc-SJIII-sun req-resp
0
100
200
300
400
500
600
10000 15000 20000 25000 30000 35000 40000 45000
Latency us
N(t
)
1024 bytes pc-SJIII-sun req-resp
0
100
200
300
400
500
600
10000 15000 20000 25000 30000 35000 40000 45000
Latency usN
(t)
1500 bytes pc-SJIII-sun req-resp
0100200300400500600700
10000 15000 20000 25000 30000 35000 40000 45000
Latency us
N(t
)
512 bytes pc-SJIII-sun req-resp
0
100
200
300
400
500
600
10000 15000 20000 25000 30000 35000 40000 45000
Latency us
N(t
)
64 bytes pc-MC req-resp
0
100
200
300
400
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
gw
-me
tro
0
500
1000
1500
2000
2500
N(t
) g
w-m
c
1400 bytes pc-MC req-resp
0
100
200
300
400
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
gw
-me
tro
0
500
1000
1500
2000
N(t
) g
w-m
c
256 bytes pc-MC req-resp
0
100
200
300
400
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
gw
-me
tro
0
500
1000
1500
2000
2500
N(t
) g
w-m
c
512 bytes pc-MC req-resp
0
100
200
300
400
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
gw
-me
tro
0
500
1000
1500
2000
2500
3000
N(t
) g
w-m
c1024 bytes pc-MC req-resp
0
100
200
300
400
0 5000 10000 15000 20000 25000 30000
Latency us
N(t
)
gw
-me
tro
0
500
1000
1500
2000
2500N
(t)
gw
-mc
Pink: LAN Blue: MAN-SJ CORE Pink: Man-RAL MB Blue: Man-RAL ~5 s nice exponential ~ 2 ms 4.5 -> 6.7 ms
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
The Grid: Throughput on SuperJANET
Manc – UCL 10 Mbit Manc – Glasgow
0
2
4
6
8
10
12
0 200 400 600 800 1000 1200 1400
Transmit Time/packet us
Re
cv W
ire
ra
te M
bit
s/s
100 bytes 200 bytes400 bytes 800 bytes600 bytes 1000 bytes1200 bytes 1400 bytes
0
5
10
15
20
25
30
35
40
45
50
0 200 400 600 800 1000 1200 1400
Transmit Time/packet us
Re
cv W
ire
Ra
te M
bit
/s
100 bytes 200 bytes400 bytes 600 bytes800 bytes 1000 bytes1200 bytes 1400 bytes
Manc – RAL 24 Mbit link Manc – RAL 2 Mbit Managed BW
0
5
10
15
20
25
30
35
0 500 1000 1500 2000 2500
Transmit Time/packet us
Rec
v W
ire
Rat
e in
clud
ing
ATM
Mbi
t/s
100 bytes 200 bytes
400 bytes 600 bytes
800 bytes 1000 bytes
1200 bytes 1400 bytes
0
0.5
1
1.5
2
2.5
0 500 1000 1500 2000 2500 3000Transmit Time/packet us
Re
cv W
ire
Rat
e in
clu
din
g A
TM
Mb
it/s
100 bytes 200 bytes
400 bytes 600 bytes
800 bytes 1000 bytes
1200 bytes 1400 bytes
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
The Grid: Multi-stream ftp Oxford to SLACSingle stream 8 ftp streams ~ 0.8 single stream
John Macallister PPNCG Oxford
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
The Grid: Multi-stream ftp SLAC to Oxford
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET4
SuperJANET4 Topology based on Worldcom Trident Network
–Tariff: distance independent–SLA: Fix problems not refunds
Backbone initially SDH based
2.5 Gbit (STM-16)Gigabit capacity with growth built in 10 Gbit in 2001,
20 Gbit in 2002Link existing Regional Networks Migration to DWDM during 2001Technology review in 2003
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET4: Backbone and Access links
Worldcom supply the transmission:–SDH–Circuit protection – fallback in 50 ms
UKERNA overlay the IP:–Packet over SDH–Core PoP IP routers at Worldcom–Backbone Access Router at MANs–Routers managed by UKERNA NOSC
Access Link Fibers:–Installed by Worldcom–Large MAN 2.5 Gbit -> 10-20 Gbit–Medium MAN 622 Mbit -> 2.5 Gbit
4 node DWDM development net
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET4: Worldcom High Level Design
DX DX
DX
DX
DX
DX
DX
DX
DX
DX
DX
DX
DX
DX
DX
DX
Manchester
London
EdinburghGlasgow
Warrington
Newcastle
Leeds
Leicester
PortsmouthExeter
Bristol
Reading
Birmingham
Cardiff
Nottingham
Greenwich
Cambridge
Belfast
STM16
STM 4
STM leased
Dark Fibre
STM 64 WC
Metro DWDM
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET4: Routing
CISCO 12016 routers from LogicalCore PoP
–Responsible for switching packets–No “edge” functions – congestion avoidance–Symmetric L3 routing – eBGP peering to BAR–Packet over SDH
Backbone
C-PoP
SDH ADM
Regional MAN
BAR
AccessLinks
eBGPPeering
Backbone Access Router–Provide filtering – drop if src. not from MAN–Traffic management – classify packets–Interface to MANs with Gigabit Ethernet
Multicast–Integral part of SuperJANET4 – not tunneled –Run sparse-mode PIM
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET4: MAN Development
Support for Router and Link upgradesIncrease:
–Capacity 1 Gbit now 10-20 Gbit in 2 years–Resilience – 99.95 % uptime–Reach – additional sites & schools
NetNorthWest as an example
NNW
SuperJANET4 BAR
Manchester
Liverpool
Preston
SDH
Gigabit Ethernet
SDH Trunk
SDH trunks 622 Mbit, will be increased with traffic load
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET4: Network Services
Services co-located at CPoPs or BARs–Provides High BW access
Centrally managed by UKERNA–Remote console access–Remote power management–Network and access control provided
Provider responsible for service not networkExamples:
–DNS servers–www caches–usnet servers–Video servers - content from academic / external–Videoconference H.323 MCUs
Backbone
Regional MAN
BAR
C-PoP
AccessLinks
Server
SDH ADM
Server
Server
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
SuperJANET4: Project UpdateRollout Status:
–Backbone in place at Worldcom C-PoPs – SDH resilience shown.–Routers at Logical UK, acceptance in progress, delivery 15 Nov 00–Manchester-ULCC traffic live on 5 Dec 00–All sites Using SuperJANET4 by 31 Mar 01
Network Users–HE sites – e.g. Universities already on SuperJANET III–FE sites – e.g. Colleges almost all connected at 2 Mbit–Schools – Further study in progress
Challenges–Quality of Service
•Priority traffic – packet marking & WRED
•Jitter sensitive traffic H.323 – Low latency queuing MDRR
–MPLS•VPNs
R. Hughes-Jones Advanced Networks Workshop Montreal November 2000
Conclusion & Status
Network Links to US and Europe are OK (at the moment !)Peering with Abilene, ESnet and CANARIE OK, interesting effects with CAR – WRED !Considerable UK interest in the Internet2 tests Grid: active in several areas including networksSuperJANET4 rollout is on target