Optimising ASP/ISP Interconnections

Panos GevrosUniversity of Cambridge

Computer Laboratorypanos.gevros@cl.cam.ac.uk

TAPAS project meeting - Dortmund, Germany

10 February 2003

outlineApplication & Internet Service Providers issues

• similarities and differences in their operations, optimisation goals

The network environment• topology, domains, routing, interconnection types, performance

implications

Interconnection choice : a common ISP & ASP problem:• Selecting which ISP(s) to interconnect with (involves facility location

and ISP assessment aspects)

Network design : an ISP-specific problem: • delay analysis and capacity provisioning

background & basic facts

• For ASP/ISP network performance is crucial for delivering high quality services (offer outsourcing, move bits around).

• In network engineering there are two basic philosophies for introducing high performance (low delay, high bandwidth) Internet services, namely :  

• QoS / service differentiation i.e. giving preferential treatment to selected traffic flows using router mechanisms and/or new service models (Intserv, Diffserv).

• bandwidth provisioning i.e. provide enough bandwidth so that the most stringent delay requirements are being met without differentiation between users.

• It is unlikely that a new QoS service model will replace the existing best-effort one

• For some time now the major players in the area base their operations on intelligent decisions about provisioning and interconnection of their networks

• However, the landscape in the area is far from clear…

Comparing ASP and ISP operations

• An ASP should • connect to ISPs so that application service performance, fulfils the terms

specified in the SLAs it has with its customers (in the absence of SLAs the perceived performance should be considered satisfactory).

• establish presence in network places which are “close” to target customers (same metropolitan area, exchange etc.)

• The difference between an ISP and an ASP is that the ISP operation is generally more broad,

• ISP customers are not only end-users, but also other network providers, • the ISP may act as a customer w.r.t some ISPs and as a provider w.r.t to others

• An ISP and an ASP are similar in that their operations depends on the relationships they establish with other ISPs.

• An ISP in many cases operates as an ASP by bundling with the network connectivity offering, infrastructure services like DNS, Mail Relay, firewall, web-caching etc.

• The ISP operation goals will become clear after we discuss the interconnection environment.

The interconnection environment

• Macroscopic view of the networkAbstract view as 3-tier hierarchy of providers

Backbone (Tier1) (e.g UUNET, C&W, Sprint)

Regional (Tier2) (national backbones, part leased from Tier1’s)

Access (Tier3) (e.g. AOL)

• Brief description of BGP routing• Types of relationships between ISPs

transit (“customer – provider”)

peering

• The role of Internet Exchanges (IX-es)

Basic definitions• Internet : as an interconnection of Autonomous Systems (ASes): • Autonomous System (AS): a collection of networks under a single

administrative authority distinguished by AS number (e.g an ISP is associated with one AS number).

• Backbone: a network of geographically dispersed POPs (Points of Presence) interconnected with high capacity links, operated by a single administrative authority.

• Point of Presence (POP) : a physical location where a collection of routers belonging to a single provider are used for the interconnection of its customers to the wide-area backbone.

• Route : an association between a destination network (network prefix) in the Internet and a next hop router towards that destination. (~112,000 IP subnets in the global Internet routing table).

• BGP : Border Gateway Protocol, routing protocol operating at the borders of the providers’ networks (between neighbor AS-es) its operation determines the routes (paths taken by the traffic flows).

A (very) brief description of BGP Operation (1/)

• BGP (Border Gateway Protocol) is a path vector protocol.

De-facto standard inter-domain routing protocol

• Operation: in-bound exchange of Route Advertisements between border routers

• Route advertisement (rtadv) consists of • Destination network prefix (e.g. 192.168.0.0/16),

• next hop router IP address,

• AS path info (list of AS#)

• If a border router of AS-x sends a route advertisement for network N to a border router of AS-y this implies that AS-x accepts to forward packets with destination address an address in N on behalf of AS-y.

A (very) brief description of BGP Operation (2/)

• AS-path info has dual functionality,• Loop detection• Route selection (shorter paths are preferred)

• Inter-domain traffic engineering with BGP filters • Input filters select which rtadv will be accepted (example policy: accept a

route advertisements if and only if the AS#s in the AS path info belong to “trusted” AS-es)

• Output filters selects which routes are being advertised to its neighbors

• Decision process: applies (configurable) route selection criteria • AS-path length, med, eBGP>iBGP, for tie-break keep oldest route)

• Selected route is installed in forwarding table (for each packet does longest match prefix of its destination address and indicates outgoing interface).

Transit and Peering : definitions (1/)

• Route advertisements are advertisements of value being offered and they are at the heart of the contracts between ISPs. Depending on the routes exchanged between two neighbor ISPs there are two types of (bilateral) relationship:

• Transit is a “customer-provider” relationship between two ISPs in which one ISP (the provider) sells to the other ISP (the customer) connectivity to all its destinations that are available in its routing table.

• Peering is the relationship in which the two ISPs mutually provide connectivity to their customers (only) without any money exchange.

ISP C

ISP 1

Customers

Suppose ISP - A is buying transit from ISP-D (denoted as A ->D) then ISP-D has the obligation of carrying A’s traffic to/from any Internet destination/source.

Let also ISP-A has a peering relationship with ISP-B (A <p> B) and ISP-B in turn peers with ISP-C (B <p> C).

Peering is a non-transitive relationship (i.e the routes of C’s customers are visible to B but they do not get propagated further to A, so traffic originating from a customer of A destined to a customer of C will not be routed via B instead it will be routed via ISP-D from which ISP-A buys transit. In the same manner ISP-A’s routes do not get propagated to ISP-C.

ISP B

ISP A

Transit and Peering: example (2/)

Transit and Peering: economic facts (1/)

• Value and money flow in opposite directions• All ISPs (except the Tier-1s) have to buy transit• Transit fees constitute the main operational costs.

• Selling transit is necessary and desirable – (increases the number of your customers and they are the ones who

pay)

• However, ISP cannot survive by re-selling transit alone, – (the ISP becomes the unnecessary middleman which the customers

bypass and connect directly to its upstream)

• Can be a non-starter, depending on selling transit for value growth, while value is a pre-requisite for attracting customers and selling transit …

Transit and Peering: economic facts (2/)

• Reason why peering becomes important: • it is an economic optimisation over buying transit, reduces

average per-bit costs.

• creates value growth (increases the amount of bandwidth the ISP has to sell to its customers)

• Parties in a peering relationship perceive that the cost of peering (fixed costs plus carrying peers traffic) is offset

• Tier-1s tend to peer only between themselves, • do not want to commoditise IP – and prefer to compete on the

basis of “better service”

• do not have incentives to improve the performance of smaller ISPs enabling them to compete against them in the market.

Peering (+)s • Reduces transit costs

• buying transit costs 10ths or 100ths thousands $$/month

• Peering is low cost, provides direct traffic exchange, reduces the load on expensive transit links.

• Reduces latency (end-to-end delay)• Direct interconnection to a competitor ISP nearby provides lower

delay compared to traversing several transit providers.

• Increases demand for bandwidth• High packet loss rates and high delays reduce bandwidth

consumption (adaptive TCP!) and therefore revenues. This can be a problem (reducing revenues) with usage based charging.

Peering (-)s

• Peering is not free– Router ports, circuits (to the exchange), man hours, cost money, lengthy

negotiations

• Assymetric traffic– When Traffic ratio across peering boundary (outbound/inbound) exceeds

certain limits then the peering may no longer be seen as beneficial to one of the parties involved (in practice ratios up to 4:1 can be found and still be considered acceptable).

• Lack of SLAs – Peering relationships unlike “customer” relationships are not followed by SLA

“guarantees” for reliability, rapid repair of problems, or penalty when a connection performs poorly.

– Side-effect: determining the performance of an end-to-end path becomes unpredictable when the path traverses several peering boundaries (public internet exchanges) which tend to be congested.

Internet Exchanges (IXes) (1/2)

• Exchange is essentially the “marketplace” where ISPs peer or buy/sell transit (first IXes were only for peering).

• IX-es offer cost savings by aggregating traffic from many neighbours on one link (the link into the exchange). (downside increased risk due to a single point of failure)

• Today IXes tend to differentiate towards transit-only or peering only.

Internet Exchanges (2/2)

• An IX it should satisfy certain criteria for ISP participation: access issues, operational, cost, reputation, viability, neutral operation, opportunities (for selling transit or for peering)

– Neutrally operated• not aligned with specific carriers or ISPs which may impose restrictions on the

participants as to where they should be buying access circuits from etc.

– The cost of using the exchange• Rack fees, port fees,cross connect fees, installation fees

– Reputation / Credibility / financial viability• Will the exchange continue to exist in the future

• ISPs tend to be “risk averse” ; prefer established, well-populated exchanges (and are willing to pay an “insurance” premium, instead of landing in an untested IX)

• ISP min-conf :presence two transit and one peering IX-es

Service Provider Interconnection decisions

• ASP should access the network so that its customers perceive a service that is fast and reliable.

• ISPs (Tier-2,3) face a somewhat similar problem.

• Commonality : the importance of location it determines the quality of

So the questions faced by both service providers (ISPs/ASPs ) are

I. to which locations, POPs, should we buy links into (“where should we land capacity “),

II. which other ISPs should we interconnect with,

III. under which terms we should be interconnecting with a certain ISP.

No–single correct answer,– depend on provider; customer base, growth plans, budget limitations, (regulatory

framework in which it operates) etc.

General principles apply (usually)

The ASP interconnection

• Depending on the services offered by the ASP there are different strategies,

1. Co-locate close to the customers2. Select the same access ISPs as the target customer group3. Act as an advisor, assess performance/connectivity offered by

different ISPs to the ASP site -- advise customers as to which ISPs provide “better” access the service.

• Before choosing an ISP the ASP should evaluate the candidate, criteria involve

• Network capacity• number of peering arrangements• information about peering policies (link capacities, upgrade

procedures, presence of SLAs)

• Improve connectivity, reduce risk• multihoming (use >1 upstream ISPs)• use redundancy (have multiple links to the same ISP, possibly

using BGP policies for load balancing).

ASP interconnection

ISP

ISP

ISP

ISP

ISPISP

ISP

ISP

ISP

ASP

customer

customer

solution 1: strategic co-location

ISP

ISP

ISP

ISP

ISP

ISP

ISP

ASP

customerASP

ASP

(possibly) VPN

solution 2 :common access ISPs

ISP

ISP

ISP

ISP

ISP

ISP

ISP

ASP

customer

solution 3 : issue ISP recommendations

ISP

ISP

ISP

ISP

ISP

ISP

ISP

ASP

customers

ISP interconnection (1)

• Interconnection decisions based on intuition and higher-level business considerations (strategic partners, alliances), often corporate ties often override technical justification.

• On the technical side the real AS path is selected by BGP using shortest path (myopic - ignores amount of traffic and cost – there is space for optimisation there).

• Decisions at this point important for the viability of the business• Pattern : ISP min-conf :presence two transit and one peering IX-es• transit usually straightforward

– (cash issue, small number of players)

• peering more complicated (reduce traffic on expensive transit links– identify targets (IX-es, ISPs) – assesment, (monitoring (i/o) traffic flows, Netflow data from routers)– negotiation, (peering policy of the identified “target” determine the outcome)– Key Question : “How much traffic are we sending to or through the AS’s that

participate in the exchange?”

ISP interconnection (2)

• Criterion: Exchanged Traffic Volume between AS-es, distinguish between

– Transit traffic passes through an AS– Terminating traffic sinks in an AS

• create AS rankings based on “transit/terminating traffic scores”.

• Assign weights to the AS-es based on their distance from the destination subnets.

• Find those AS that appears to be central in most popular paths and try to peer with those AS!

ISP interconnection (3)

• More tangible evaluation criteria– The ownership of the links ; determines the quality of the offered service.

When the ISP owns the links it has full control over potential problems.– The physical and link layer technologies deployed – compatibility of the

technologies and their intrinsic value (i.e. state-of-the art versus outdated transmission technologies) can determine the reliability of the service, how easy/fast it will be to schedule future capacity upgrades (e.g light-up more fiber is easy) etc.

– The diversity of the physical routes – physical path separation inside the core provider’s network is important so that customers’ mission critical traffic is not vulnerable to outages or single points of failure. Sometimes the logical topology maps can be deceptive as they may show route diversity but the underlying physical links are not diverse. Also in the metropolitan area where the links terminate there should be POPs in different physical locations (buildings).

Decision-making framework (1)

• The minimal ISP reference model– buy Transit from two other ISPs– Peer with as many ISPs as possible (meaningful)– sell transit to customers

• Create an optimal “portfolio” of peering and transit interconnections.

Decision-making framework (2)

Inputs: • transit traffic vector• terminating traffic vector• cost of peering with an AS (infinite if not available)• cost of buying transit from an AS• cost of peering with an exchange

Algorithm for determining the threshold for participating in an exchange

Peering provides redundancy reduces connectivity risk• Prob[ dest-i unreachable ]• weight dest-i based on traffic transitting through or

terminating to it

Decision-making framework (3)

Criteria for the selection of an upstream provider: • geographic proximity (access circuit cost)

• Network reliability (uptime)

• Pricing (unit bandwidth cost)

• Customer support

• Value added services (etc..)

Optimisation Algorithm (most likely heuristic)

Network design

Two problems:– capacity assignment for delay minimisation – assumes delay analysis

Delay Analysis (1/)

• Model: N-nodes, M-links, capacities Ci , i=1,..,M measured in

packets/sec (assumption all packets have the same length 1/)

• Poisson process with mean rate i,j (packets/sec) external

traffic entering (and leaving – i.e. no termination inside the network)

N

j

M

kjk

1 1

M

ii

1

node -j node -knetwork

j k

jki

Delay Analysis (2/)

• Link in the network is modelled as an M/M/1 “server with a queue” the arrival process of the packets at link-i is I

• Let Zj,k is the expected delay for a packet entering the providers

network at node j and exiting at node k, • Let T be the average (expected) delay a packet will experience

using the network

kj

N

j

N

k

ji ZT ,1 1

,

Decompose T in sum of link delays Ti be the average delay a

packet incurs at link-i (waiting time in the queue plus transmission time).

Delay Analysis (3/)

Where link delays Ti are given by standard M/M/1 results

(arrival rate i and service times 1/Ci )

i

M

i

iTT

1

iii CT

1

Capacity Assignment (1/)

• The provider knows • the traffic (rate bit/s) on each of the links { i }, • the topology and the routing,

• The goal is to minimise the mean delay T incurred by a packet crossing the network where the minimisation happens with respect to the link capacities { Ci } subject to total cost constraints.

• The cost incurred by the network provider to install a link of capacity C i is given by a function d(Ci) (linear, logarithmic, follow a power law etc. depending on the physical length of the link, the technology etc).

• Obviously Ci > i / and the optimisation problem becomes

DCdts

CTM

ii

i

)(..

)(min

1

Capacity Assignment (2/)

• The problem is solved for linear cost function d( Ci ) = di Ci and

the yields

Where Dx = D - i di/ is the excess cost (i / is the

minimum required amount) and n is the average path length defined as n = /

2

1

][

M

i

ii

x

d

D

nT

Summary

• ISPs and ASPs face similar interconnection problems,

• new economic models for the ISP industry focus on user control on the edges and less concern about extracting rent from content hosting.

• There has been no systematic way to approach these interconnection problems

• There is need for new tools and models to assist decision making in the peering/transit arena, facility location

• These will help the ISP to manage both their interconnection costs and the performance of their network then they will be able to “standardise” offered services and possibly provide meaningful SLAs to the customers.

http://www.cl.cam.ac.uk/~pg281/tapas/