Policy towards Research Policy towards Research NetworkingNetworking

David Williams, CERNDavid Williams, CERN

TERENA-NORDUnet Conference, LundTERENA-NORDUnet Conference, Lund

Thursday 10 June 1999Thursday 10 June 1999

David.O.Williams@cern.chDavid.O.Williams@cern.ch

Slides: http://nicewww.cern.ch/~davidw/public/Lund.pptSlides: http://nicewww.cern.ch/~davidw/public/Lund.ppt

DisclaimersDisclaimers

MinefieldMinefield

Very personal view of how to negotiate itVery personal view of how to negotiate it

Few conclusions, rather initiate discussionsFew conclusions, rather initiate discussions

Me, not my organisationMe, not my organisation

OutlineOutline

A few words on applicationsA few words on applications Basic ideas concerning the organisational Basic ideas concerning the organisational

structure of networksstructure of networks Some examplesSome examples Problems to fit things togetherProblems to fit things together Technology evolutionTechnology evolution Can there be any sensible policies?Can there be any sensible policies? Conclusions??Conclusions??

A few words on applicationsA few words on applications

Applications (1/2)Applications (1/2)

E-mailE-mail– Low volume (but grows with fancier attachments)Low volume (but grows with fancier attachments)– Needs worldwide connectivity (asynchronous)Needs worldwide connectivity (asynchronous)

General information accessGeneral information access– Bulk of conventional Internet traffic (say 75%)Bulk of conventional Internet traffic (say 75%)– Bulk of info served from USA (mainly from CA)Bulk of info served from USA (mainly from CA)– Needs worldwide connectivity (weak realtime)Needs worldwide connectivity (weak realtime)

Data sharing and accessData sharing and access– Important for scienceImportant for science– Speed and reliability matter (a lot)Speed and reliability matter (a lot)– Reliable off-peak xfers could be scheduledReliable off-peak xfers could be scheduled– Needs worldwide connectivity (weak realtime)Needs worldwide connectivity (weak realtime)

Applications (2/2)Applications (2/2)

Remote logonRemote logon– Hard to eliminate (many people hoped to…)Hard to eliminate (many people hoped to…)– Needs worldwide connectivityNeeds worldwide connectivity– And human to machine performance and And human to machine performance and

reliability - “as if the connection were local”reliability - “as if the connection were local” Collaborative workingCollaborative working

– Potentially vital/very valuable for organisations Potentially vital/very valuable for organisations which depend heavily on international workingwhich depend heavily on international working

– Needs worldwide connectivityNeeds worldwide connectivity– Audio and video and shared whiteboard Audio and video and shared whiteboard

aspects require excellent performance and aspects require excellent performance and reliability (tough RT constraints)reliability (tough RT constraints)

Basic ideas concerning the Basic ideas concerning the organisational structure of organisational structure of

networksnetworks

What is a network - What is a network - organisationally?organisationally?

EITHEREITHER– a business which offers to sell network services a business which offers to sell network services

to customers, in accordance with some more or to customers, in accordance with some more or less well-defined specifications and a price listless well-defined specifications and a price list

OROR– a group of users who decide to join together to a group of users who decide to join together to

acquire network services, either by joint acquire network services, either by joint buying, or by assuming joint responsibility for buying, or by assuming joint responsibility for building those services.building those services.

USER GROUPING vs SUPPLIER POWER??USER GROUPING vs SUPPLIER POWER??

Four main (organisational) Four main (organisational) characteristics of a netcharacteristics of a net

Who can connect?Who can connect? What are the goals and what traffic is What are the goals and what traffic is

allowed?allowed? With which other nets does it directly With which other nets does it directly

exchange traffic?exchange traffic? For which other nets does it carry transit For which other nets does it carry transit

traffic?traffic?

Network A

Network B

Network C

Traffic exchange directly between A and B

Traffic exchange directly between B and C

Traffic exchange between A and C transits over B

Goals?Users allowed?Traffic allowed?

Who can connect?Who can connect?

Either anyone who pays Either anyone who pays Or people who belong to “the user group”Or people who belong to “the user group” In this case, do they belongIn this case, do they belong

– BECAUSEBECAUSE most of their traffic is with other most of their traffic is with other group members, so network performance (RTT, group members, so network performance (RTT, bandwidth, reliability) between group members bandwidth, reliability) between group members is important?is important?

– OR BECAUSEOR BECAUSE, though little of their traffic is , though little of their traffic is with other group members, but they can all with other group members, but they can all acquire a more cost-effective network by acquire a more cost-effective network by acting together?acting together?

Goals and allowed traffic?Goals and allowed traffic?

Making a profit vs price-performance vs fair Making a profit vs price-performance vs fair competitioncompetition– Goal of ISP is to make a profitGoal of ISP is to make a profit– Goal of group net is cost-effective service for membersGoal of group net is cost-effective service for members– Normally there are some (national) rules governing Normally there are some (national) rules governing

interactions between group nets and ISPsinteractions between group nets and ISPs Group goals vs group non-goalsGroup goals vs group non-goals

– Traffic profile should meet the agreed goals of groupTraffic profile should meet the agreed goals of group– Depending on the funding authority and the applicable Depending on the funding authority and the applicable

rules, the profile is sometimes quite tightly definedrules, the profile is sometimes quite tightly defined– Major source of “problems” can come from rules for Major source of “problems” can come from rules for

carrying transit trafficcarrying transit traffic

With which other nets do you With which other nets do you directly exchange traffic? (1/2)directly exchange traffic? (1/2)

Remember that “worldwide connectivity” is a Remember that “worldwide connectivity” is a fundamental objective of most Internet fundamental objective of most Internet applicationsapplications

So group nets typically connect to all nets with So group nets typically connect to all nets with similar goals in other countriessimilar goals in other countries– Certainly if traffic can be exchanged for “free”Certainly if traffic can be exchanged for “free”– But, trans-oceanic bandwidth is (inherently) But, trans-oceanic bandwidth is (inherently)

expensive, so cost-sharing is an issueexpensive, so cost-sharing is an issue Group nets also typically connect to one or more Group nets also typically connect to one or more

commercial nets to handle their commercial commercial nets to handle their commercial traffictraffic– Choosing a supplier is a price-performance decisionChoosing a supplier is a price-performance decision

With which other nets do you With which other nets do you directly exchange traffic? (2/2)directly exchange traffic? (2/2)

Commercial networks connectCommercial networks connect– ““downwards” to ISPs who are willing to pay their downwards” to ISPs who are willing to pay their

chargescharges– and “upwards” or “sideways” to ISPs which they and “upwards” or “sideways” to ISPs which they

can make arrangements withcan make arrangements with

For which other nets do you For which other nets do you carry transit traffic?carry transit traffic?

If you are a big ISP you carry transit traffic If you are a big ISP you carry transit traffic for small ISPs as part of your normal for small ISPs as part of your normal businessbusiness

If you are a big ISP you carry transit traffic If you are a big ISP you carry transit traffic for other big ISPs that you have reached for other big ISPs that you have reached arrangements witharrangements with

If you are a group net you typically carry If you are a group net you typically carry transit traffic when you think that the transit traffic when you think that the arrangement will be mutually beneficialarrangement will be mutually beneficial

Some practical examplesSome practical examples

NRNs amd TEN-155NRNs amd TEN-155

(US) Agency networks(US) Agency networks

Internet-2 / AbileneInternet-2 / Abilene

vBNSvBNS

Commercial networksCommercial networks

Specialised networksSpecialised networks

National academic & research nets National academic & research nets (1/2)(1/2)

Who can belong?Who can belong?– Decided on a case-by-case basis in each Decided on a case-by-case basis in each

countrycountry– Always for research universitiesAlways for research universities– Nearly always for research labs (but not in US)Nearly always for research labs (but not in US)– What about non-research universities?What about non-research universities?– What about secondary schools?What about secondary schools?– What about primary schools?What about primary schools?– What about libraries?What about libraries?– What about local government offices?What about local government offices?– What about commercial companies “spun-off” What about commercial companies “spun-off”

from universities or research centres?from universities or research centres?

National academic & research nets National academic & research nets (2/2)(2/2)

Goals and allowed trafficGoals and allowed traffic– Good service for groupGood service for group– So, good site-to-site performance is importantSo, good site-to-site performance is important– General (commodity) information access General (commodity) information access

important tooimportant too Traffic exchange?Traffic exchange?

– Other nation’s A&R and related nets (agency nets, Other nation’s A&R and related nets (agency nets, TEN-155, etc)TEN-155, etc)

– One or more commercial ISPsOne or more commercial ISPs Transit traffic?Transit traffic?

– Not muchNot much

TEN-155TEN-155

Who can belong?Who can belong?– European NRNs + ….European NRNs + ….

Goals and allowed trafficGoals and allowed traffic– Exchange among members according to AUPExchange among members according to AUP– US traffic handling for some membersUS traffic handling for some members– (MBS)(MBS)

Traffic exchange?Traffic exchange?– See belowSee below

Transit traffic?Transit traffic?– Carries traffic between NRNs and non-European Carries traffic between NRNs and non-European

(Japan, USA?, …) nets handling A&R-related traffic, (Japan, USA?, …) nets handling A&R-related traffic, where agreements are in placewhere agreements are in place

Agency networks (1/2)Agency networks (1/2)

BackgroundBackground– In USA several agencies are funded by federal In USA several agencies are funded by federal

governmentgovernment– And universities are either private or funded by And universities are either private or funded by

state governmentsstate governments– Strong funding separationStrong funding separation

Who can belong?Who can belong?– Agency funded labs and projectsAgency funded labs and projects

Goals and allowed trafficGoals and allowed traffic– Defined by agency missionDefined by agency mission

Agency networks (2/2)Agency networks (2/2)

Traffic exchange?Traffic exchange?– To get to sites important for agency missionTo get to sites important for agency mission

Transit traffic?Transit traffic?– Hardly ever (I think), on the grounds that its Hardly ever (I think), on the grounds that its

not their job to carry that trafficnot their job to carry that traffic– One example is that ESnet does not carry One example is that ESnet does not carry

transit traffic from (European) NRNs to US transit traffic from (European) NRNs to US universities universities

Internet-2 / Abilene (1/2)Internet-2 / Abilene (1/2)

Who can belong?Who can belong?– UCAID members (for the Internet-2 aspect)UCAID members (for the Internet-2 aspect)– Who have also paid up to use Abilene (maybe Who have also paid up to use Abilene (maybe

with help of NSF grant)with help of NSF grant) Goals and allowed trafficGoals and allowed traffic

– High performance connectivity between US High performance connectivity between US research universitiesresearch universities

– So, in principle, no “commodity” trafficSo, in principle, no “commodity” traffic

Internet-2 / Abilene (2/2)Internet-2 / Abilene (2/2)

Traffic exchange?Traffic exchange?– With vBNS and US agency netsWith vBNS and US agency nets– With international NRNs With international NRNs – According to local (not Internet-2) decision, with According to local (not Internet-2) decision, with

any ISP or net present at the GigaPoPany ISP or net present at the GigaPoP Transit traffic?Transit traffic?

– Only according to I-2 exchange agreements as Only according to I-2 exchange agreements as discussed in previous pointdiscussed in previous point

vBNS (1/2)vBNS (1/2) Who can belong?Who can belong?

– NSF grant holdersNSF grant holders– Who will normally (not necessarily) be UCAID Who will normally (not necessarily) be UCAID

membersmembers Goals and allowed trafficGoals and allowed traffic

– Support of NSF collaborative projects requiring Support of NSF collaborative projects requiring high performance connectivityhigh performance connectivity

– In principle, no traffic unrelated to those projectsIn principle, no traffic unrelated to those projects– A rather tight AUPA rather tight AUP

vBNS (2/2)vBNS (2/2) Traffic exchange?Traffic exchange?

– With Abilene and agency netsWith Abilene and agency nets– With international NRNs connected via STAR-TAPWith international NRNs connected via STAR-TAP– According to local decision, with any ISP or net According to local decision, with any ISP or net

present at the GigaPoPpresent at the GigaPoP Transit traffic?Transit traffic?

– Only according to I-2 exchange agreements as Only according to I-2 exchange agreements as discussed in previous pointdiscussed in previous point

Commercial networksCommercial networks

Who can belong?Who can belong?– Anyone who paysAnyone who pays

Goals and allowed trafficGoals and allowed traffic– To make a profitTo make a profit– No (few) restrictions on trafficNo (few) restrictions on traffic– Rather strong flow to/from USRather strong flow to/from US

Traffic exchange?Traffic exchange?– Decided on commercial groundsDecided on commercial grounds

Transit traffic?Transit traffic?– Decided on commercial groundsDecided on commercial grounds

Specialised networks, such as Specialised networks, such as ANXANX

Who can belong?Who can belong?– Members of the clubMembers of the club

Goals and allowed trafficGoals and allowed traffic– Run a highly performant Internet for (US?) Run a highly performant Internet for (US?)

automobile companies and their suppliersautomobile companies and their suppliers– By qualifying ISPs and exchanges, and By qualifying ISPs and exchanges, and

monitoring their performancemonitoring their performance Traffic exchange?Traffic exchange?

– Gateway to public InternetGateway to public Internet Transit traffic?Transit traffic?

– NoNo

Problems to fit things togetherProblems to fit things together

CongestionCongestion

Packet lossPacket loss

Exchange pointsExchange points

SatSat SunSunSunSun

MonMon

TueTue

WedWed

ThursThurs

FriFri

Daily packet loss structure on a congested route

50% peaks 8 loss-free hours at nightFrom ~01.00 to 09.00 CET

FNAL-Brown connectivity - fixed in June/July ‘98

(Some) things do improve with time!!

Catastrophic overloadCatastrophic overload

Getting QoS/DS into production will be a very Getting QoS/DS into production will be a very welcome advancewelcome advance

But I suspect that no amount of QoS/DS will But I suspect that no amount of QoS/DS will help on links which are catastrophically help on links which are catastrophically overloadedoverloaded

Roads, cars, railways, phones are typically in Roads, cars, railways, phones are typically in use for less than 10% of the time, or deliver use for less than 10% of the time, or deliver only some 10% of their theoretical bandwidth, only some 10% of their theoretical bandwidth, when measured on a 168 hours/week, 52 when measured on a 168 hours/week, 52 weeks/year basisweeks/year basis

Exchange pointsExchange points

The government structures are very The government structures are very different in Europe and in the USA. That different in Europe and in the USA. That leads to network goals and rules for leads to network goals and rules for allowed traffic which don’t match wellallowed traffic which don’t match well

Deciding at which points to exchange Deciding at which points to exchange traffic is not so obvioustraffic is not so obvious

And the cost of getting there may be And the cost of getting there may be (very) high(very) high

Leads to difficult discussions on cost Leads to difficult discussions on cost sharingsharing

Technology evolutionTechnology evolution

Again, personal viewsAgain, personal views

Since last year in Dresden (1/3)Since last year in Dresden (1/3)

Quick move to TEN-155 in EuropeQuick move to TEN-155 in Europe Most national nets in Europe deploy or Most national nets in Europe deploy or

plan 155 Mbps or faster backbonesplan 155 Mbps or faster backbones Internet-2 has become a reality with vBNS Internet-2 has become a reality with vBNS

(622 Mbps, soon to move to 2.5 Gbps)(622 Mbps, soon to move to 2.5 Gbps) I-2 starts to deploy a multiple backbone I-2 starts to deploy a multiple backbone

architecture - Abilene (2.5 Gbps) plus architecture - Abilene (2.5 Gbps) plus vBNS, using GigaPoP as a basic vBNS, using GigaPoP as a basic componentcomponent

QoS (still) coming…..QoS (still) coming…..

Since last year in Dresden (2/3)Since last year in Dresden (2/3)

Deregulation in Europe reduced some prices Deregulation in Europe reduced some prices quite quicklyquite quickly

N. Atlantic pricing also fell a lotN. Atlantic pricing also fell a lot Concept of an (all) Optical Internet became Concept of an (all) Optical Internet became

concrete, most obviously in Canarie-3concrete, most obviously in Canarie-3 Based on HDWDM technology, headed Based on HDWDM technology, headed

towards 100s or 1000s of towards 100s or 1000s of ss May completely change how you build MANs May completely change how you build MANs

(80-100 km without (80-100 km without anyany amplification) amplification) Gigabit Ethernet is winning many Gigabit Ethernet is winning many

commercial battles as the line protocolcommercial battles as the line protocol

Since last year in Dresden (3/3)Since last year in Dresden (3/3)

But ATM (VCs) more widely deployed as But ATM (VCs) more widely deployed as backbone technology and that helped backbone technology and that helped quite a bit X-At by reducing hop countsquite a bit X-At by reducing hop counts

And in some critical Internet ExchangesAnd in some critical Internet Exchanges

Initial MONARC work convinces me that we Initial MONARC work convinces me that we are working in a very challenging area of are working in a very challenging area of distributed computing with really huge distributed computing with really huge data volumes where few companies are data volumes where few companies are activeactive

Can there be any sensible Can there be any sensible policies?policies?

Worry more about InterconnectsWorry more about Interconnects

Worry less about commodity Worry less about commodity traffictraffic

Improve our technical Improve our technical understandingunderstanding

Concentrate on services for A&RConcentrate on services for A&R

InterconnectsInterconnects

Because of poor competitive situation we Because of poor competitive situation we have tended to worry mainly about line have tended to worry mainly about line costs in Europecosts in Europe

We should worry at least as much about the We should worry at least as much about the places where we interconnectplaces where we interconnect

All A&R sites in an (extended) city should All A&R sites in an (extended) city should work together to create a GigaPoPwork together to create a GigaPoP

GigaPoPsGigaPoPs

Where you separate out your traffic (and if Where you separate out your traffic (and if that’s hard, let’s work on it)that’s hard, let’s work on it)

And send different types of traffic to And send different types of traffic to different nets (including ISPs)different nets (including ISPs)

And encourage competitive presence of And encourage competitive presence of suppliers of bandwidthsuppliers of bandwidth

Including competitive suppliers of Including competitive suppliers of s and s and of (dark) fibreof (dark) fibre

Commodity traffic (1/2)Commodity traffic (1/2)

How wise is it that European NRNs carry long-How wise is it that European NRNs carry long-distance commodity traffic?distance commodity traffic?

10 years ago, this was essential, since ISPs 10 years ago, this was essential, since ISPs could not provide the service (at a sensible could not provide the service (at a sensible price)price)

My inclination tells me NRNs should try to My inclination tells me NRNs should try to offload commodity traffic as quickly as offload commodity traffic as quickly as possiblepossible

Provided it can be done at reasonable price-Provided it can be done at reasonable price-performanceperformance

NRNs for bulk-buying compared to NRNs as NRNs for bulk-buying compared to NRNs as ISPs for their local communityISPs for their local community

Commodity traffic (2/2)Commodity traffic (2/2)

They would still handle all of the traffic for They would still handle all of the traffic for NRN customers, just hand commercial part NRN customers, just hand commercial part off to an ISP offering the best price-off to an ISP offering the best price-performance service as soon as feasibleperformance service as soon as feasible

Otherwise the distinction between NRNs Otherwise the distinction between NRNs and ISPs gets very blurredand ISPs gets very blurred

And the NRNs are vital for maintaining a And the NRNs are vital for maintaining a sensible A&R network inside each countrysensible A&R network inside each country

Wider understanding of Internet Wider understanding of Internet traffictraffic

Instrumentation of the Internet and Instrumentation of the Internet and understanding of its performance has vastly understanding of its performance has vastly improved in the past ~18 monthsimproved in the past ~18 months

Know a lot about RTT and packet loss rates Know a lot about RTT and packet loss rates on many routeson many routes

Know much less about the integral traffic Know much less about the integral traffic flows and the engineering of Interconnectsflows and the engineering of Interconnects

Work to do thereWork to do there

Services for A&RServices for A&R

Another global trend is that ISPs start to offer Another global trend is that ISPs start to offer many (optional) services to their customers.many (optional) services to their customers.

Such as e-mail hosting, Web hosting, home Such as e-mail hosting, Web hosting, home directory support (backup etc), …, and I directory support (backup etc), …, and I expect this trend to grow. Internet Service expect this trend to grow. Internet Service Provider ==> Application SP; ISP to ASP.Provider ==> Application SP; ISP to ASP.

NRNs natural customers are a quite well-NRNs natural customers are a quite well-defined population with some clear defined population with some clear application service needs.application service needs.

Should make sure that they are well taken Should make sure that they are well taken care ofcare of

Conclusions??Conclusions??

Just defined what I regard as sensible Just defined what I regard as sensible policiespolicies

What about the European dimension?What about the European dimension?

The European dimension (1/2)The European dimension (1/2) There are very important roles for the There are very important roles for the

NRNs, DANTE, TERENA and the EU to all NRNs, DANTE, TERENA and the EU to all play in not losing the momentum created play in not losing the momentum created by the success of TEN-155by the success of TEN-155

If each country had a set of well-run If each country had a set of well-run GigaPoPsGigaPoPs

Which were interconnected nationallyWhich were interconnected nationally It would help enormously in creating a It would help enormously in creating a

competitive situation for interconnecting competitive situation for interconnecting them on a pan-European basisthem on a pan-European basis

The European dimension (2/2)The European dimension (2/2) The EU needs to be especially careful of The EU needs to be especially careful of

thinking that you can ignore the A&R thinking that you can ignore the A&R community and build an e-commerce community and build an e-commerce infrastructure in glorious isolationinfrastructure in glorious isolation

Funding 2-3 labs to study Internet protocols Funding 2-3 labs to study Internet protocols and performance in the real world would be and performance in the real world would be really very usefulreally very useful

As would getting to understand how much As would getting to understand how much Europe could benefit from a very broad Europe could benefit from a very broad collaboration between suppliers, industry, collaboration between suppliers, industry, commerce, academia and research labs commerce, academia and research labs