ICFA Standing Committee on ICFA Standing Committee on Interregional Connectivity (SCIC) Interregional Connectivity (SCIC)
Harvey B. NewmanHarvey B. Newman California Institute of Technology California Institute of Technology
ICFA Seminar, CERNICFA Seminar, CERNOctober 9, 2002October 9, 2002
IIIICCCCFFFFAAAA aaaannnndddd GGGGlllloooobbbbaaaallll NNNNeeeettttwwwwoooorrrrkkkkssss ffffoooorrrrHHHHEEEENNNNPPPP
IIIICCCCFFFFAAAA aaaannnndddd GGGGlllloooobbbbaaaallll NNNNeeeettttwwwwoooorrrrkkkkssss ffffoooorrrrIIIICCCCFFFFAAAA aaaannnndddd GGGGlllloooobbbbaaaallll NNNNeeeettttwwwwoooorrrrkkkkssss ffffoooorrrrHHHHEEEENNNNPPPPHHHHEEEENNNNPPPP
��National and International Networks, with sufficientNational and International Networks, with sufficientcapacity and capability, are essential forcapacity and capability, are essential for➨➨The daily conduct of collaborative work in bothThe daily conduct of collaborative work in both
experiment and theory experiment and theory➨➨Detector development, construction on a global scale;Detector development, construction on a global scale;
Data analysis involving physicists from all world regionsData analysis involving physicists from all world regions➨➨The conception, design and implementation ofThe conception, design and implementation of
next generation facilities as next generation facilities as ““global networksglobal networks””➨➨The formation of worldwide collaborationsThe formation of worldwide collaborations
�� ““Collaborations on this scale would never haveCollaborations on this scale would never havebeen attempted, if they could not rely on excellentbeen attempted, if they could not rely on excellentnetworksnetworks””
ICFA and International NetworkingICFA and International Networking
�� ICFA Statement on Communications in IntICFA Statement on Communications in Int’’l HEPl HEPCollaborations of October 17, 1996Collaborations of October 17, 1996 See See http://http://www.fnal.gov/directorate/icfa/icfa_communicaes.htmlwww.fnal.gov/directorate/icfa/icfa_communicaes.html
““ICFA urges that all countries and institutions wishingICFA urges that all countries and institutions wishingto participate even more effectively and fully into participate even more effectively and fully ininternational HEP Collaborations should:international HEP Collaborations should:�� Review their operating methods to ensure theyReview their operating methods to ensure they
are fully adapted to remote participation are fully adapted to remote participation
�� Strive to provide the necessary communications Strive to provide the necessary communications facilities and adequate international bandwidth facilities and adequate international bandwidth””
ICFA Network Task Force: 1998Bandwidth Requirements Projection
(Mbps)
ICFA Network Task Force: 1998Bandwidth Requirements Projection
(Mbps)1998 2000 2005
BW Utilized Per Physicist(and Peak BW Used)
0.05 - 0.25(0.5 - 2)
0.2 – 2(2-10)
0.8 – 10(10 – 100)
BW Utilized by a UniversityGroup
0.25 - 10 1.5 - 45 34 - 622
BW to a Home Laboratory OrRegional Center
1.5 - 45 34 - 155 622 - 5000
BW to a Central LaboratoryHousing One or More MajorExperiments
34 - 155 155 - 622 2500 - 10000
BW on a Transoceanic Link 1.5 - 20 34 - 155 622 - 5000
100100––1000 X Bandwidth Increase Foreseen for 1998-20051000 X Bandwidth Increase Foreseen for 1998-2005See the ICFA-NTF Requirements Report:See the ICFA-NTF Requirements Report:
http://l3www.cern.ch/~newman/icfareq98.htmlhttp://l3www.cern.ch/~newman/icfareq98.html
NTF
ICFA Standing Committee onInterregional Connectivity (SCIC)
ICFA Standing Committee onInterregional Connectivity (SCIC)
�� Created by ICFA in July 1998 in Vancouver ; Following ICFA-NTFCreated by ICFA in July 1998 in Vancouver ; Following ICFA-NTF�� CHARGE:CHARGE:
Make recommendations to ICFA concerning theMake recommendations to ICFA concerning theconnectivity between connectivity between the Americasthe Americas, Asia and Europe, Asia and Europe
➨➨ As part of the process of developing theseAs part of the process of developing theserecommendations, the committee shouldrecommendations, the committee should�� Monitor trafficMonitor traffic�� Keep track of technology developmentsKeep track of technology developments�� Periodically review forecasts of futurePeriodically review forecasts of future
bandwidth needs, andbandwidth needs, and�� Provide early warning of potential problemsProvide early warning of potential problems
�� Create subcommittees when necessary to meet the chargeCreate subcommittees when necessary to meet the charge�� The chair of the committee should report to ICFA once perThe chair of the committee should report to ICFA once per
year, at its joint meeting with laboratory directors (Feb. 2003)year, at its joint meeting with laboratory directors (Feb. 2003)�� Representatives: Major labs, ECFA, ACFA, NA Users, S. AmericaRepresentatives: Major labs, ECFA, ACFA, NA Users, S. America
ICFA-SCIC Core MembershipICFA-SCIC Core Membership
�� Representatives from major HEPRepresentatives from major HEPlaboratories:laboratories:
Manuel Manuel DelfinoDelfino (CERN)(CERN)
to W. Von to W. Von RuedenRueden Michael Ernst Michael Ernst (DESY)(DESY) Matthias Matthias KasemannKasemann (FNAL) (FNAL) Yukio Karita Yukio Karita (KEK)(KEK) Richard Mount Richard Mount (SLAC)(SLAC)
�� User RepresentativesUser Representatives Richard Hughes-Jones (UK)Richard Hughes-Jones (UK) Harvey Newman Harvey Newman (USA)(USA)
Dean Dean KarlenKarlen (Canada)(Canada)
�� For Russia:For Russia: SlavaSlava IlyinIlyin (MSU)(MSU)
�� ECFA representatives: ECFA representatives:
Frederico Ruggieri (INFNFrederico Ruggieri (INFNFrascati),Frascati),Denis Linglin (IN2P3, Lyon)Denis Linglin (IN2P3, Lyon)
�� ACFA representatives: ACFA representatives:
RongshengRongsheng XuXu (IHEP Beijing) (IHEP Beijing)
HwanBaeHwanBae Park Park((KyungpookKyungpook Nat Nat’’llUniversity)University)
�� For South America: For South America: Sergio F. NovaesSergio F. Novaes(University of Sao Paulo)(University of Sao Paulo)
LHC Data Grid HierarchyLHC Data Grid Hierarchy
Tier 1
Tier2 Center
Online System
CERN 700k SI95 ~1 PB Disk;Tape Robot
FNAL: 200kSI95; 600 TBIN2P3 Center INFN CenterRAL Center
InstituteInstituteInstituteInstitute~0.25TIPS
Workstations
~100-400MBytes/sec
2.5 Gbps
100 - 1000Mbits/sec
Tens of Petabytes by 2007-8.An Exabyte within ~5 Years later.Physics data cache
~PByte/sec
~2.5 Gbits/sec
Tier2 CenterTier2 CenterTier2 Center~2.5 Gbps
Tier 0 +1
Tier 3
Tier 4
Tier2 CenterTier 2
Experiment
CERN/Outside Resource Ratio ~1:2Tier0/(ΣΣΣΣ Tier1)/(ΣΣΣΣ Tier2) ~1:1:1
Transatlantic Net WG (HN, L. Price) Bandwidth Requirements (2001) [*] Transatlantic Net WG (HN, L. Price) Bandwidth Requirements (2001) [*]
� 2001 2002 2003 2004 2005 2006
CMS 100 200 300 600 800 2500
ATLAS 50 100 300 600 800 2500
BaBar 300 600 1100 1600 2300 3000
CDF 100 300 400 2000 3000 6000
D0 400 1600 2400 3200 6400 8000
BTeV 20 40 100 200 300 500
DESY 100 180 210 240 270 300
CERN BW
155-310
622 2500 5000 10000 20000
[*] [*] Installed BW. Maximum Link Occupancy 50% AssumedInstalled BW. Maximum Link Occupancy 50% AssumedSee http://See http://gate.hep.anl.gov/lprice/TANgate.hep.anl.gov/lprice/TAN
History – One large Research Site
Projections: 0.5 to 24 Tbps by ~2012
Much of the Traffic:SLAC ���� IN2P3/RAL/INFN;
via ESnet+France;Abilene+CERN
ICFA SCIC Meetings[*] and TopicsICFA SCIC Meetings[*] and Topics
�� Focus on the Digital Divide This Year Focus on the Digital Divide This Year➨➨ Identification of problem areas; work on ways to improve Identification of problem areas; work on ways to improve
�� Network Status and Upgrade Plans in Each Country Network Status and Upgrade Plans in Each Country➨➨ Performance (Throughput) Evolution in Each Country, Performance (Throughput) Evolution in Each Country,
and Transatlantic and Transatlantic�� Performance Monitoring World-Overview Performance Monitoring World-Overview (Les Cottrell, IEPM Project) (Les Cottrell, IEPM Project)�� Specific Technical Topics (Examples): Specific Technical Topics (Examples):➨➨ Bulk transfer, Bulk transfer, QoSQoS, Collaborative Systems, Security, VOIP, Collaborative Systems, Security, VOIP
�� Preparation of Reports to ICFA (Lab Directors Preparation of Reports to ICFA (Lab Directors’’ Meetings) Meetings)➨➨ Last Report: World Network Status and Outlook - Feb. 2002 Last Report: World Network Status and Outlook - Feb. 2002➨➨ Next Report: Next Report: Digital Divide, +Digital Divide, + Monitoring, Advanced Monitoring, Advanced
Technologies, and Requirements Evolution Technologies, and Requirements Evolution –– Feb. 2003 Feb. 2003
[*] Six Meetings in 2002; Next at KEK In December 13.[*] Six Meetings in 2002; Next at KEK In December 13.
SCIC Sub-Committees SCIC Sub-CommitteesWeb Page Web Page http://http://cern.chcern.ch/ICFA-SCIC//ICFA-SCIC/
�� Monitoring:Monitoring: Les Cottrell Les Cottrell((http://www.slac.stanford.edu/xorg/icfa/scic-netmonhttp://www.slac.stanford.edu/xorg/icfa/scic-netmon))With Richard Hughes-Jones (Manchester), With Richard Hughes-Jones (Manchester), Sergio Sergio NovaesNovaes (Sao (SaoPaolo); Paolo); SergeiSergei BerezhnevBerezhnev (RUHEP), (RUHEP), FukukoFukuko Yuasa (KEK), Daniel Yuasa (KEK), DanielDavidsDavids (CERN), Sylvain (CERN), Sylvain RavotRavot (Caltech), Shawn McKee (Michigan) (Caltech), Shawn McKee (Michigan)
�� Advanced Technologies:Advanced Technologies: Richard Hughes-Jones w Richard Hughes-Jones with ith VladimirVladimirKorenkovKorenkov (JINR, (JINR, DubnaDubna), Olivier ), Olivier Martin(CERNMartin(CERN), ), Harvey NewmanHarvey Newman
�� End-to-end Connectivity:End-to-end Connectivity: Richard Mount (SLAC) Richard Mount (SLAC)➨➨ With Michael Ernst, Denis With Michael Ernst, Denis LinglinLinglin, , AlexandreAlexandre SztajnbergSztajnberg (Rio, Brazil) (Rio, Brazil)
�� The Digital Divide:The Digital Divide: Alberto Santoro (Rio, Brazil) Alberto Santoro (Rio, Brazil)➨➨ With With SlavaSlava IlyinIlyin, Yukio , Yukio KaritaKarita, David O. Williams, David O. Williams➨➨ Also Also DongchulDongchul Son (Korea), Son (Korea), HafeezHafeez HooraniHoorani (Pakistan), (Pakistan),
SunandaSunanda BanerjeeBanerjee (India), Vicky White (FNAL) (India), Vicky White (FNAL)�� Key Requirements:Key Requirements: Harvey Newman Harvey Newman
➨➨ Also Charlie Young (SLAC)Also Charlie Young (SLAC)
Network Progress andIssues for Major Experiments
Network Progress andIssues for Major Experiments
�� Major R&E Networks have weathered the economic Major R&E Networks have weathered the economic ““stormstorm””�� Backbones & major links advancing rapidly to 10 Backbones & major links advancing rapidly to 10 GbpsGbps range range
➨➨ ““GbpsGbps”” end-to-end throughput data flows have been end-to-end throughput data flows have been tested; will be in production soon (in 1-2 years) tested; will be in production soon (in 1-2 years)
�� Network advances are changing the view of the netNetwork advances are changing the view of the net’’s roless roles➨➨ Progress to and beyond 10 Progress to and beyond 10 GbpsGbps within next few years within next few years➨➨ Likely to have a profound impact on the experimentsLikely to have a profound impact on the experiments’’
Computing Models, and bandwidth requirementsComputing Models, and bandwidth requirements➨➨ More dynamic view: dynamic path provisioning;More dynamic view: dynamic path provisioning;
GB to TB data transactionsGB to TB data transactions�� Net R&D Driven by Advanced integrated applications, suchNet R&D Driven by Advanced integrated applications, such
as Data Grids, that rely on seamless LAN and WAN operationas Data Grids, that rely on seamless LAN and WAN operation➨➨ With reliable, quantifiable (monitored), high performanceWith reliable, quantifiable (monitored), high performance
➜➜ All of the above will further open the Digital Divide chasm.All of the above will further open the Digital Divide chasm.We need to take actionWe need to take action
Signs of the Times: UncertaintyBut No Change in Outlook
Signs of the Times: UncertaintyBut No Change in Outlook
�� While While Key Providers are in BankruptcyKey Providers are in Bankruptcy➨➨ KPNQwestKPNQwest, , TeleglobeTeleglobe,, Global Crossing, FLAG; Global Crossing, FLAG; WorldcomWorldcom
�� A&R Nets and Major Links Have Continued Their Advance; A&R Nets and Major Links Have Continued Their Advance;Switched Vendors where neededSwitched Vendors where needed➨➨ E.g. T-Systems (Deutsche Telecom) for US-CERNE.g. T-Systems (Deutsche Telecom) for US-CERN
�� Telecom Market Outlook Telecom Market Outlook➨➨ Good pricing continues, in Good pricing continues, in ““open competitionopen competition”” situations situations➨➨ MCI/MCI/WorldcomWorldcom networks continue to operate: networks continue to operate:
20 M customers in US; UK academic & research network20 M customers in US; UK academic & research network➨➨ Level3 market share advance; perhaps also Tyco, GXLevel3 market share advance; perhaps also Tyco, GX
�� Link Donations !: NY-Amsterdam OC192 (Tyco to IEEAF) Link Donations !: NY-Amsterdam OC192 (Tyco to IEEAF)�� Overall: Strong Outlook in R&E Nets Overall: Strong Outlook in R&E Nets➨➨ Regional Progress, and Visions; E.g. IWIRE;Regional Progress, and Visions; E.g. IWIRE;
CALREN: CALREN: ““1 1 GbpsGbps to Every Californian by 2010 to Every Californian by 2010””
ICFA SCIC: R&E Backbone andInternational Link Progress
ICFA SCIC: R&E Backbone andInternational Link Progress
�� GEANT Pan-European BackboneGEANT Pan-European Backbone ( (http://www.dante.net/geanthttp://www.dante.net/geant))➨➨ Now interconnects >31 countries; many trunks Now interconnects >31 countries; many trunks 2.5 and 10 2.5 and 10 GbpsGbps
�� UKUK: : SuperJANETSuperJANET Core at Core at 10 10 GbpsGbps➨➨ 2.5 2.5 GbpsGbps NY-London, with 622 Mbps to NY-London, with 622 Mbps to ESnetESnet and Abilene and Abilene
�� France (IN2P3): France (IN2P3): 2.5 2.5 GbpsGbps RENATER backbone by mid-October RENATER backbone by mid-October➨➨ Lyon-CERN Link being Upgraded to 1 Lyon-CERN Link being Upgraded to 1 GbpsGbps Ethernet Ethernet➨➨ Proposal for dark fiber to CERN by end 2003Proposal for dark fiber to CERN by end 2003
�� SuperSINETSuperSINET (Japan): (Japan): 10 10 GbpsGbps IP and 10 IP and 10 GbpsGbps Wavelength Wavelength Core Core➨➨ Tokyo to NY Links: Now 2 X 2.5 Tokyo to NY Links: Now 2 X 2.5 GbpsGbps; Need to get to Starlight; Need to get to Starlight
�� CA*net4 (Canada):CA*net4 (Canada): Interconnect customer-owned dark fiber Interconnect customer-owned dark fiber nets across Canada at nets across Canada at 10 10 GbpsGbps, started July 2002, started July 2002➨➨ ““Lambda-GridsLambda-Grids”” by ~2004-5 by ~2004-5
�� GWIN (Germany):GWIN (Germany): 2.5 2.5 GbpsGbps Core Core; Connect to US at 2 X 2.5 ; Connect to US at 2 X 2.5 GbpsGbps;;Support for SILK Project: Satellite links to FSU RepublicsSupport for SILK Project: Satellite links to FSU Republics
�� Russia: Russia: Now 155 Mbps links to MoscowNow 155 Mbps links to Moscow➨➨ Moscow-Starlight Link to Moscow-Starlight Link to 155 Mbps155 Mbps (US NSF + Russia Support) (US NSF + Russia Support)➨➨ Moscow-Stockholm Link; connect to GEANT at Moscow-Stockholm Link; connect to GEANT at 155 Mbps155 Mbps
R&E Backbone and Int’l Link ProgressR&E Backbone and Int’l Link Progress
�� Abilene (Internet2) UpgradeAbilene (Internet2) Upgrade from 2.5 to from 2.5 to 10 10 GbpsGbps Underway Underway➨➨ Encourage high throughput use for targeted applicationsEncourage high throughput use for targeted applications
�� ESNET: UpgradeESNET: Upgrade to 10 to 10 GbpsGbps ““NowNow””�� US-CERNUS-CERN➨➨ to 622 Mbps in August; Move to STARLIGHTto 622 Mbps in August; Move to STARLIGHT➨➨ 2.5G Research Triangle from August 20022.5G Research Triangle from August 2002
STARLIGHT-CERN-NL; to 10G in 2003STARLIGHT-CERN-NL; to 10G in 2003�� SLAC + IN2P3 (SLAC + IN2P3 (BaBarBaBar))➨➨ To ~100 Mbps throughput over US-CERN and To ~100 Mbps throughput over US-CERN and RenaterRenater links links➨➨ 600 Mbps Throughput is 600 Mbps Throughput is BaBarBaBar Target for this Year Target for this Year
(with (with ESnetESnet and IN2P3 Link Upgrades) and IN2P3 Link Upgrades)�� FNALFNAL➨➨ ESnetESnet Link Upgrade to 622 Mbps Link Upgrade to 622 Mbps➨➨ Plans for dark fiber to STARLIGHT, planned for this FallPlans for dark fiber to STARLIGHT, planned for this Fall
��NY-Amsterdam NY-Amsterdam Donation Donation from Tyco, from Tyco, September 2002: September 2002:Arranged by IEEAF: Arranged by IEEAF: 622 Gbps+10 622 Gbps+10 GbpsGbps Research Wavelength Research Wavelength
NNNNaaaattttiiiioooonnnnaaaallll RRRReeeesssseeeeaaaarrrrcccchhhh NNNNeeeettttwwwwoooorrrrkkkkssssiiiinnnn JJJJaaaappppaaaannnn
NNNNaaaattttiiiioooonnnnaaaallll RRRReeeesssseeeeaaaarrrrcccchhhh NNNNeeeettttwwwwoooorrrrkkkkssssiiiinnnn JJJJaaaappppaaaannnn
SuperSINETSuperSINET➨➨ Started operation January 4, 2002 Started operation January 4, 2002➨➨ Support for 5 important areas: Support for 5 important areas:
HEP,HEP, Genetics, Genetics, NanoNano-Technology,-Technology,Space/Astronomy, Space/Astronomy, GRIDsGRIDs
➨➨ Provides 10 Provides 10 λλλλλλλλ’’s:s:�� 10 10 GbpsGbps IP connection IP connection�� Direct Direct intersiteintersite GbEGbE links links�� 7 Univ. Connected; 7 Univ. Connected; Add 3 More this Month Add 3 More this Month
HEPnetHEPnet-J-J➨➨ Reconstructed with Reconstructed with MPLS-VPN in MPLS-VPN in SuperSINETSuperSINET
Proposal: Two Proposal: Two TransPacificTransPacific 2.5 2.5 GbpsGbps Wavelengths, and Wavelengths, andJapan-CERN Grid Japan-CERN Grid TestbedTestbed by ~2003 by ~2003
Tokyo
Osaka
Nagoya
Internet
Osaka U
Kyoto U
ICRKyoto-U
Nagoya U
NIFS
NIG
KEK
Tohoku U
IMSU-Tokyo
NAO
U Tokyo
NII Hitot.
NII Chiba
IP
WDM path
IP router
OXC
ISAS
National R&E Network ExampleGermany: DFN Transatlantic Connectivity 2002
STM 4
STM 16
STM 1
6
�� FSU Connections via satellite:FSU Connections via satellite:Yerevan, Minsk, Yerevan, Minsk, AlmatyAlmaty, Baikal, Baikal➨➨ Speeds of 32 - 512 kbps Speeds of 32 - 512 kbps
�� SILK Project (2002): NATO fundingSILK Project (2002): NATO funding➨➨ Links to Caucasus and Central Links to Caucasus and Central Asia (8 Countries) Asia (8 Countries)➨➨In 2001-2 64-512 kbpsIn 2001-2 64-512 kbps➨➨Proposed VSAT for 10-50 X BW:Proposed VSAT for 10-50 X BW: NATO + State Funding NATO + State Funding
�� 2 X OC48: NY-Hamburg 2 X OC48: NY-Hamburgand NY-Frankfurtand NY-Frankfurt
�� Direct Peering to Abilene (US) and Direct Peering to Abilene (US) and CanarieCanarie (Canada) (Canada)
��UCAID will adding another 2 OC48UCAID will adding another 2 OC48’’s;s;in a Proposed Global Terabitin a Proposed Global TerabitResearch Network (GTRN)Research Network (GTRN)
*
Also see http://www-iepm.slac.stanford.edu/monitoring/bulk/;and the Internet2 E2E Initiative: http://www.internet2.edu/e2e
Max. Sustained TCP Throughputon Transatlantic Links
Max. Sustained TCP Throughputon Transatlantic Links
�� 8/10/01 105 Mbps reached with 30 Streams: SLAC-IN2P38/10/01 105 Mbps reached with 30 Streams: SLAC-IN2P3�� 9/1/01 102 Mbps in One Stream: CIT-CERN 9/1/01 102 Mbps in One Stream: CIT-CERN�� 11/5/01 125 Mbps in One Stream (modified kernel): CIT-CERN 11/5/01 125 Mbps in One Stream (modified kernel): CIT-CERN�� 1/09/02 190 Mbps for One stream shared on 2 155 Mbps links 1/09/02 190 Mbps for One stream shared on 2 155 Mbps links�� 3/11/02 120 Mbps 3/11/02 120 Mbps Disk-to-DiskDisk-to-Disk with One Stream on 155 Mbps with One Stream on 155 Mbps
link (Chicago-CERN)link (Chicago-CERN)�� 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams�� 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel) 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel)�� 9/02 850 Mbps Chicago-CERN One Stream on OC48 (mod. 9/02 850 Mbps Chicago-CERN One Stream on OC48 (mod. KernetKernet)) 1350+ (1900+) Mbps with 2 (3) Streams on 2 (3) 1350+ (1900+) Mbps with 2 (3) Streams on 2 (3) GbEGbE Ports Ports
US-CERN DataTAG Link Testswith Grid TCP (S.Ravot et al.): 9/02
US-CERN DataTAG Link Testswith Grid TCP (S.Ravot et al.): 9/02
3 Streams;3 GbE Ports3 Streams;3 GbE Ports
1 to 3Streams;
1-3 GbE Ports
1 to 3Streams;
1-3 GbE Ports
1 32
Next: Tests to 10 Gbps; 1 GB/sec disk to disk
TOE;CPU;
Protocol
TOE;CPU;
Protocol
IIIIEEEEPPPPMMMM:::: PPPPiiiinnnnggggEEEERRRRDDDDeeeeppppllllooooyyyymmmmeeeennnnttttIIIIEEEEPPPPMMMM:::: PPPPiiiinnnnggggEEEERRRRDDDDeeeeppppllllooooyyyymmmmeeeennnntttt
�� Measurements fromMeasurements from➨➨ 34 monitors in 14 countries34 monitors in 14 countries➨➨ Over 600 remote hostsOver 600 remote hosts➨➨ Over 72 countriesOver 72 countries➨➨ Over 3300 monitor-remote site pairsOver 3300 monitor-remote site pairs➨➨ Measurements go back to Jan-95Measurements go back to Jan-95➨➨ Reports on RTT, loss, Reports on RTT, loss, reachabilityreachability,,
jitter, reorders, duplicates jitter, reorders, duplicates ……
�� Countries monitoredCountries monitored➨➨Contain 78% ofContain 78% of
world populationworld population➨➨ 99% of online users99% of online users
of the Internetof the Internet➨➨Mainly A&R sitesMainly A&R sites
Monitoring Sites
Remote Sites
History - Throughput Quality Improvements from USBandwidth of TCP < MSS/(RTT*Sqrt(Loss)) (1)
(1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm , , MatthisMatthis,,SemkeSemke, , MahdaviMahdavi, , OttOtt, Computer Communication Review 27(3), July 1997, Computer Communication Review 27(3), July 1997
80% annualimprovementFactor ~100/8 yr
Progress: but Digital Divide is Maintained
Work on the Digital Divide:Several Perspectives
Work on the Digital Divide:Several Perspectives
�� Identify & Help Solve Technical Problems:Identify & Help Solve Technical Problems: NatNat’’l, Regional, Last 10/1/0.1 kml, Regional, Last 10/1/0.1 km
�� Inter-Regional Proposals (Example: Brazil)Inter-Regional Proposals (Example: Brazil)➨➨ US NSF Proposal (10/2002); possible EU LIS ProposalUS NSF Proposal (10/2002); possible EU LIS Proposal
�� Work on Policies and/or Pricing: Work on Policies and/or Pricing: pkpk, in, , in, brbr, , cncn, SE Europe, , SE Europe, ……➨➨ E.g. E.g. RoEduNetRoEduNet (2-6 Mbps); Pricing not so different (2-6 Mbps); Pricing not so different
from US-CERN price in 2002 for a few from US-CERN price in 2002 for a few GbpsGbps➨➨ Find Ways to work with vendors, Find Ways to work with vendors, NRENsNRENs, and/or , and/or GovGov’’tsts
�� Use Model Cases: Installation of new advanced fiberUse Model Cases: Installation of new advanced fiberinfrastructures; Convince Neighboring Countriesinfrastructures; Convince Neighboring Countries➨➨ Poland (to 5k km Fiber); Hungary; Slovakia; IrelandPoland (to 5k km Fiber); Hungary; Slovakia; Ireland
�� Exploit One-off Solutions: Exploit One-off Solutions: E.g. extend the SILK ProjectE.g. extend the SILK Project(DESY/FSU satellite links) to a SE European site(DESY/FSU satellite links) to a SE European site
�� Work with other organizations:Work with other organizations: TerenaTerena, Internet2, AMPATH,, Internet2, AMPATH,IEEAF, UN, etc. IEEAF, UN, etc. to help with technical and/or political to help with technical and/or political solsol’’nsns
GEANT
155Mbps
34Mbps 34Mbps
34Mbps
34Mbps
34MbpsAnnual Cost
> 1 MEuro
Romania: 155 Mbps to GEANT and Bucharest;Inter-City Links of 2-6 Mbps
Romania: 155 Mbps to GEANT and Bucharest;Inter-City Links of 2-6 Mbps
NREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002
NREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002
Logarithmic Scale
1k
100k
100
100M
10M
1M
10kRo
It
PlGrIr
Ukr
Hu Cz
Es
Nl
Fi
Ch
Lagging
In Transition
Leading
Advanced
RNP Brazil (to 20 Mbps)
FIU Miami from So. America (to 80 Mbps)
Note: Auger (AG), ALMA (Chile),CMS-Tier1 (Brazil)
Digital Divide ActivitiesDigital Divide Activities�� Questionnaire Being Distributed (Discuss At ICFA Meeting) Questionnaire Being Distributed (Discuss At ICFA Meeting)➨➨ CERN/IT to Assist with Web Form; Online SubmissionCERN/IT to Assist with Web Form; Online Submission
�� Plan on Project to Build HENP World Network Map; Plan on Project to Build HENP World Network Map; Updated and Maintained on a Web Site, Backed by Database: Updated and Maintained on a Web Site, Backed by Database:➨➨ Systematize and Track Needs and Status Systematize and Track Needs and Status➨➨ Information: Link Bandwidths, Utilization, Quality, Pricing, Information: Link Bandwidths, Utilization, Quality, Pricing,
Local Infrastructure, Last Mile Problems, Vendors, etc. Local Infrastructure, Last Mile Problems, Vendors, etc.➨➨ Identify Urgent Cases; Focus on Opportunities to Help Identify Urgent Cases; Focus on Opportunities to Help
�� First ICFA SCIC Workshop: Focus on the Digital DivideFirst ICFA SCIC Workshop: Focus on the Digital Divide➨➨ Target Date February 2004 in Rio de Janeiro (LISHEP) Target Date February 2004 in Rio de Janeiro (LISHEP)➨➨ Organization Meeting July 2003 Organization Meeting July 2003➨➨ Plan Statement at the WSIS, Geneva (December 2003) Plan Statement at the WSIS, Geneva (December 2003)➨➨ Install and Leave Behind a Good Network Install and Leave Behind a Good Network➨➨ Then 1 (to 2) Workshops Per Year, at Sites that Need Help Then 1 (to 2) Workshops Per Year, at Sites that Need Help
Digital Divide Sub-Committee: Questionnaire Response Extract:
34 Mbps Link toGEANT.Universitypays for Network
HEP group intends to build a type T2 or T3 Grid node, andcontribute to MC Production. Need to upgrade Network toGigabit/sec. In principle there is no access limit theNetwork now. But daily traffic load is the real limitation.
University ofCYPRUS
Will maintain a farm for Monte Carlo studies and a Tier 3Grid node; need more bandwidth
IFT-UNESPBrazil
Will have a Tier3 Grid node; Need network bandwidthupgrades at reasonable price (Telecom monopoly)
TIFR, India
Also plans for T2at NUST
In terms of bandwidth need to upgrade but no last mileconnection problem. High Prices
QAU, Pakistan
Additional bandwidth needed for HEP; now 2 Mbps for > 1000 people
JSI, Slovenia
a) LAN upgrade to 100 Mbps; b) LAN-to-WAN upgrade to 4 Mbps
UNLP, Argentina
Dedicated 2 Mbps link for HEP group Needed(now 2 X 2 Mbps for Whole Community; Not Enough)
Cinvestav,Mexico
Wait for approvedfunds to build aT3; In 2005/6 a T1
HEPGRID PROJECT presented for financial support to workon CMS; T3 then T1. Last Mile Problem: Need to reach RNPwith good routing from UERJ
UERJ, Brazil
OtherComputing / networking needs related to HEPInstitution
Groningen CarrierHotel: March 2002
“Cultivate and promotepractical solutions todelivering scalable,
universally available andequitable access to
suitable bandwidth andnecessary network
resources in support ofresearch and education
collaborations.”
http://www.ieeaf.org
CN
SG
PERTH
GHANA
BuenosAires/San
Paolo
St. Petersburg
KazakhstanUzbekistan
ChenaiNavi
Mumbai
Barcelona GreeceMD
NL
CA
Global Quilt Initiative – GMRE Initiative - 001
Global Medical Research Exchange Initiative Bio-Medicine and Health Sciences
Layer 1 – Spoke & Hub Sites
Layer 2 – Spoke & Hub Sites
Layer 3 – Spoke & Hub Sites
Propose Global Research and Education Network for Physics
Networking for HENPNetworking for HENP
�� Networks are vital for global collaboration, andNetworks are vital for global collaboration, andthe present and future of Our Field; But: the present and future of Our Field; But: Digital DivideDigital Divide
What ICFA and the HENP Community Can DoWhat ICFA and the HENP Community Can Do�� Spread the message: ICFA SCIC is there to helpSpread the message: ICFA SCIC is there to help�� Help identify and highlight specific needs (to Work On)Help identify and highlight specific needs (to Work On)
➨➨Policy problems; Last Mile problems; etc.Policy problems; Last Mile problems; etc.�� Encourage Joint programs [such as in Encourage Joint programs [such as in DESYDESY’’ss Silk project; Silk project;
Japanese links to SE Asia and China; AMPATH to So. America]Japanese links to SE Asia and China; AMPATH to So. America]➨➨ NSF & LIS Proposals: US and EU to South America NSF & LIS Proposals: US and EU to South America
�� Make direct contacts, arrange discussions with Make direct contacts, arrange discussions with govgov’’tt officials officials➨➨ ICFA SCIC is prepared to participate ICFA SCIC is prepared to participate
�� Help Start, or Get Support for Workshops on Networks (& Grids)Help Start, or Get Support for Workshops on Networks (& Grids)➨➨ Discuss & Create opportunities Discuss & Create opportunities➨➨ Encourage, help form funded programs Encourage, help form funded programs
�� Help form Regional support & training groups (requires funding)Help form Regional support & training groups (requires funding)
Networks, Grids and HENPNetworks, Grids and HENP�� Next generation 10 Next generation 10 GbpsGbps network backbones are network backbones are
almost here: in the US, Europe and Japanalmost here: in the US, Europe and Japan➨➨ First stages arriving, starting now First stages arriving, starting now
�� Major transoceanic links at 2.5 - 10 Major transoceanic links at 2.5 - 10 GbpsGbps in 2002-3 in 2002-3�� Getting high (reliable; Grid) application performanceGetting high (reliable; Grid) application performance
across networks means!across networks means!➨➨ End-to-end monitoring; a coherent approach End-to-end monitoring; a coherent approach➨➨ Getting high performance (TCP) toolkits in users Getting high performance (TCP) toolkits in users’’ hands hands➨➨ Working in concert with AMPATH, Internet2, Working in concert with AMPATH, Internet2, TerenaTerena;; DataTAGDataTAG, the Grid projects and the Global Grid Forum, the Grid projects and the Global Grid Forum
�� Network improvements are especially needed inNetwork improvements are especially needed inSE Europe, So. America; SE Asia, and Africa:SE Europe, So. America; SE Asia, and Africa:➨➨ Key Examples: India, Pakistan, China; Brazil; Romania Key Examples: India, Pakistan, China; Brazil; Romania
�� Removing regional, Last mile bottlenecks and compromisesRemoving regional, Last mile bottlenecks and compromisesin network quality are now (in all world regions)in network quality are now (in all world regions) On the critical pathOn the critical path
0
5000
10000
15000
20000
Link Bandwidth (Mbps)
BW (Mbps) 310 622 2500 5000 10000 20000
FY2001
FY2002
FY2003
FY2004
FY2005
FY2006
Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF)Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF)
�� DataTAGDataTAG 2.5 2.5 GbpsGbps Research Link since Summer 2002 Research Link since Summer 2002�� 10 10 GbpsGbps Research Link by Approx. Mid-2003 Research Link by Approx. Mid-2003
TransoceanicNetworking
Integrated withthe Abilene,
TeraGrid,Regional Nets
and ContinentalNetwork
Infrastructuresin US, Europe,
Asia, SouthAmerica
Baseline evolution typicalBaseline evolution typicalof major HENPof major HENP
links 2001-2006 links 2001-2006
History – Loss Quality (Cottrell)
�Fewer sites have very poor to dreadful performance
�More have good performance (< 1% Loss)
History – oneResearch ISP
100% growth in traffic/yr for
last 12 years
Continuous upgrades
Increase packet size (bulk
throughput apps)
Key Network Issues &Challenges
Key Network Issues &Challenges
Net Infrastructure Requirements for High ThroughputNet Infrastructure Requirements for High Throughput��Packet Loss must be ~Zero (at and below 10Packet Loss must be ~Zero (at and below 10-6-6))
�� I.e. No I.e. No ““CommodityCommodity”” networks networks�� Need to track down Need to track down uncongesteduncongested packet loss packet loss
��No Local infrastructure bottlenecksNo Local infrastructure bottlenecks�� Multiple Gigabit Ethernet Multiple Gigabit Ethernet ““clear pathsclear paths”” between between
selected host pairs are needed nowselected host pairs are needed now�� To 10 To 10 GbpsGbps Ethernet paths by 2003 or 2004 Ethernet paths by 2003 or 2004
��TCP/IP stack configuration and tuning Absolutely RequiredTCP/IP stack configuration and tuning Absolutely Required�� Large Windows; Possibly Multiple Streams Large Windows; Possibly Multiple Streams�� New Concepts of New Concepts of Fair UseFair Use Must then be Developed Must then be Developed
��Careful Router, Server, Client, Interface configurationCareful Router, Server, Client, Interface configuration�� Sufficient CPU, I/O and NIC throughput sufficient Sufficient CPU, I/O and NIC throughput sufficient
��End-to-endEnd-to-end monitoring and tracking of performance monitoring and tracking of performance��Close collaboration with local and Close collaboration with local and ““regionalregional”” network staffs network staffs
TCP Does Not Scale to the 1-10 TCP Does Not Scale to the 1-10 GbpsGbps Range Range
AssistedAssisted by by LoansLoans: Level3 (OC192) : Level3 (OC192) andand CiscoCisco (10GbE (10GbE andand 16X1GbE) 16X1GbE)
iGrid2002: OC192+OC48 iGrid2002: OC192+OC48 SetupSetupSeptemberSeptember 2002 2002
Short Term Donation from Level 3
Short Term Donation from Level 3
Argonne
Starlight-Chicago
Amsterdam
CERN
HENP Major Links: BW Roadmapin Gbps (HBN); Shown at ICHEP2002
HENP Major Links: BW Roadmapin Gbps (HBN); Shown at ICHEP2002
Year Production Experimental Remarks
2001 0.155 0.622-2.5 SONET/SDH
2002 0.622 2.5 SONET/SDH DWDM; GigE Integ.
2003 2.5 10 DWDM; 1 + 10 GigE Integration
2005 10 2-4 X 10 ???? Switch; ???? Provisioning
2007 2-4 X 10 ~10 X 10; 40 Gbps
1st Gen. ???? Grids
2009 ~10 X 10 or 1 -2 X 40
~5 X 40 or ~20-50 X 10
40 Gbps ???? Switching
2011 ~5 X 40 or
~20 X 10
~25 X 40 or ~100 X 10
2nd Gen ???? Grids Terabit Networks
2013 ~Terabit ~MultiTerabit ~Fill a Fiber
LHCnet Network : Late 2002LHCnet Network : Late 2002
Development and testsDevelopment and tests
Abilene MRENESnet STARTAPNASA
Linux PC forPerformance tests
& Monitoring
CERN -Geneva
GEANT Switch IN2P3 WHO
Linux PC forPerformance tests
& Monitoring
Caltech/DoE PoP – StarLight Chicago
622 Mbps (Prod.)622 Mbps (Prod.)
Cisco 7609CERN
Cisco 7609Caltech(DoE)
Alcatel 7770DataTAG(CERN)
Cisco 7606DataTAG(CERN)
Juniper M10DataTAG(CERN)
Cisco 7606Caltech(DoE)
Juniper M10Caltech(DoE)
Alcatel 7770DataTAG(CERN)
2.5 2.5 GbpsGbps (R&D) (R&D)
Optical Mux/DmuxAlcatel 1670
Optical Mux/DmuxAlcatel 1670
NLNLSURFnet
GENEVA
UKUKSuperJANET4
ABILENEABILENE
ESNETESNET
CALRENCALREN
ItItGARR-B
GEANT
NewYork
FrFrRenater
STAR-TAP
STARLIGHT
DataTAGDataTAG Project Project
�� EU-Solicited Project. EU-Solicited Project. CERNCERN, PPARC (UK), Amsterdam (NL), and INFN (IT);, PPARC (UK), Amsterdam (NL), and INFN (IT);and US (DOE/NSF: UIC, NWU and Caltech) partnersand US (DOE/NSF: UIC, NWU and Caltech) partners
��Main Aims:Main Aims:➨➨ Ensure maximum interoperability between US and EU Grid ProjectsEnsure maximum interoperability between US and EU Grid Projects➨➨ Transatlantic Transatlantic TestbedTestbed for advanced network research for advanced network research
�� 2.5 2.5 GbpsGbps Wavelength Triangle 7/02 (10 Wavelength Triangle 7/02 (10 GbpsGbps Triangle in 2003) Triangle in 2003)
Wave Triangle
The Rapid Pace of NetworkTechnology Advances Continues
The Rapid Pace of NetworkTechnology Advances Continues
Within the Next One to Two YearsWithin the Next One to Two Years�� 10 10 GbpsGbps Ethernet on Switches and Servers; Ethernet on Switches and Servers;
LAN/WAN integration at 10 LAN/WAN integration at 10 GbpsGbps➨➨ Intel 10 Intel 10 GbEGbE server server NICsNICs in Beta Test in Beta Test
�� 40 40 GbpsGbps Wavelengths Being Shown Wavelengths Being Shown�� NextGenNextGen Routers: 100 Routers: 100 MppsMpps forwarding engines, 4 forwarding engines, 4
and More 10 and More 10 GbpsGbps ports per Slot; Terabit/sec ports per Slot; Terabit/secbackplanes etc.backplanes etc.
�� Broadband Wireless [Multiple 3G/4G alternatives]: Broadband Wireless [Multiple 3G/4G alternatives]: the drive to defeat the last mile problem the drive to defeat the last mile problem➨➨ 802.11 802.11 abab, UWB; Mobile: CDMA2000/1xEV-DO, UWB; Mobile: CDMA2000/1xEV-DO
HENP Lambda Grids:Fibers for Physics
HENP Lambda Grids:Fibers for Physics
�� Problem: Extract Problem: Extract ““SmallSmall”” Data Subsets of 1 to 100 Terabytes from 1 Data Subsets of 1 to 100 Terabytes from 1to 1000 to 1000 PetabytePetabyte Data Stores Data Stores
�� Survivability of the HENP Global Grid System, withSurvivability of the HENP Global Grid System, withhundreds of such transactions per day (circa 2007)hundreds of such transactions per day (circa 2007)requires that each transaction be completed in arequires that each transaction be completed in arelatively short time.relatively short time.
�� Example: Take 800 Example: Take 800 secssecs to complete the transaction. Then to complete the transaction. Then Transaction Size (TB)Transaction Size (TB) Net Throughput (Net Throughput (GbpsGbps))
1 10 1 10 10 100 10 100 100 1000 (Capacity of 100 1000 (Capacity of
Fiber Today) Fiber Today)�� Summary: Providing Switching of 10 Summary: Providing Switching of 10 GbpsGbps wavelengths wavelengths
within ~3 years; and Terabit Switching within 5-7 yearswithin ~3 years; and Terabit Switching within 5-7 yearswould enable would enable ““PetascalePetascale Grids with Terabyte transactions Grids with Terabyte transactions””,,as required to fully realize the discovery potential of major HENPas required to fully realize the discovery potential of major HENPprograms, as well as other data-intensive fields.programs, as well as other data-intensive fields.
WSIS-RO: Bucharest Nov. 7-9WSIS is 12/03 & 05 (Geneva & Tunis)
WSIS-RO: Bucharest Nov. 7-9WSIS is 12/03 & 05 (Geneva & Tunis)
�� The Regional Ministerial Conference, based on the UN/ECEThe Regional Ministerial Conference, based on the UN/ECEmembership, is in preparation of the World Summit on themembership, is in preparation of the World Summit on theInformation Society - Bucharest, 7-9 November 2002Information Society - Bucharest, 7-9 November 2002
�� The UN General Assembly adopted in 2001 a resolution endorsingThe UN General Assembly adopted in 2001 a resolution endorsingthe organization of the the organization of the World Summit on the Information SocietyWorld Summit on the Information Society(WSIS), to be convened under the patronage of the United Nations(WSIS), to be convened under the patronage of the United NationsSecretary-General, Secretary-General, KofiKofi AnnanAnnan, with the International, with the InternationalTelecommunication Union taking the lead role in its preparationTelecommunication Union taking the lead role in its preparationalong with interested UN organizations and the host countries.along with interested UN organizations and the host countries.The Summit will offer a unique opportunity for the worldThe Summit will offer a unique opportunity for the worldcommunity to discuss and give shape to the Information Societycommunity to discuss and give shape to the Information Societyby bringing together governments, international organizations,by bringing together governments, international organizations,private sector and civil society. In preparation for this Summitprivate sector and civil society. In preparation for this Summitregional conferences are organized to develop specific proposals.regional conferences are organized to develop specific proposals.
WSIS-RO: Role of New Technologiesin Developing the Information SocietyWSIS-RO: Role of New Technologies
in Developing the Information Society
�� Tony Hey, UK Core E-Science ProgramTony Hey, UK Core E-Science Program
�� Peter Peter ZangliZangli, European Commission, European Commission
�� Hans Hoffmann, CERN Scientific and TT DirectorHans Hoffmann, CERN Scientific and TT Director
�� FabrizioFabrizio GagliardiGagliardi: EDG Project Leader: EDG Project Leader
�� William Johnston, Lawrence Berkeley LabWilliam Johnston, Lawrence Berkeley Lab
�� Dave Reese or Tom West, CENIC, Pacific and NatDave Reese or Tom West, CENIC, Pacific and Nat’’l Light Raill Light Rail
�� Don Riley or Ed Don Riley or Ed FontegrossiFontegrossi, IEEAF, IEEAF
�� Harvey Newman, CaltechHarvey Newman, Caltech
Building Petascale Global Grids:Implications for Society
Building Petascale Global Grids:Implications for Society
�� Meeting the challenges of Meeting the challenges of Petabyte-to-ExabytePetabyte-to-ExabyteGrids, and Gigabit-to-Terabit Networks, willGrids, and Gigabit-to-Terabit Networks, will transform research in science and engineering transform research in science and engineering
�� These developments will create the first trulyThese developments will create the first truly global virtual organizations (GVO) global virtual organizations (GVO)
�� If these developments are successful this couldIf these developments are successful this couldlead to profound advances in industry, commercelead to profound advances in industry, commerceand society at largeand society at large
➨➨ By changing the relationship between people By changing the relationship between people and and ““persistentpersistent”” information in their daily lives information in their daily lives
➨➨ Within the next five to ten years Within the next five to ten years