New Directions in Peering for Tier-2 and Content

Providers

Jeb R. Linton

Staff Network Engineer, EarthLink

jrlinton@ieee.org

Introduction and Acknowledgements Who am I and what am I doing here?

Acknowledgements

Topics to be Covered

Peering Motivations and Economics The Cost of Peering - History Disruptions - What do they mean?

New Low Transit Costs New Tier-1 Peering Initiative Metropolitan Multipoint Private Peering

FUD (Teaser: What Happens Next?)

Peering Motivations and Economics

The Role of Peering

To Large Tier-1s Mission-Critical

To the Rest of Us Used to Decrease Cost of Transit Used to Increase Quality of Routes Usually not mission-critical *

* Your Mileage May Vary

The Cost of Peering - History

Peering Cost – the Legacy NAPs

2-3 years ago, cost justification for NAP peering was easy based on the high cost of Transit

See Bill Norton’s whitepaper: A Business Case for ISP Peering for the origin and explanation of this method of cost justification. Bill’s graphing method will be used throughout.

Transit Cost vs. NAP Peering - Before

$-

$200

$400

$600

$800

$1,000

$1,200

$1,400

10 16 25 40 65 100

160

250

400

650

1000

Bandwidth (Mbps)

Co

st

Old Transit

NAP #1

NAP #2

NAP #3

Peering Cost – the Legacy NAPs, cont’d

Disruption #1: Rapid drop in Transit cost makes legacy NAPs much less desirable

Transit Cost vs. NAP Peering - After

$-

$200

$400

$600

$800

$1,000

$1,200

$1,400

10 16 25 40 65 100

160

250

400

650

1000

Bandwidth (Mbps)

Co

st

New Transit

NAP #1

NAP #2

NAP #3

Peering Cost – Private Lines as Supplement

Evolution: NAP Peering is supplemented with Private Peering.

Private Line to a Single Peer

$-

$100

$200

$300

$400

$500

$600$700

$800

10 25 65 160

400

1000

2500

6500

Bandwidth (Mbps)

Co

stNew Transit

NAP #1

NAP #2

NAP #3

Single PL

Peering Cost –Private Lines for Many Peers

Problem: Private Peering to many Peers can be difficult because of the repeated expenditure and low circuit utilization efficiency

Private Lines to Four Peers

$-

$100

$200

$300

$400

$500

$600

$700

$800

Bandwidth (Mbps)

Co

st

New Transit

NAP #1

NAP #2

NAP #3

Priv. Lines

Peering Cost – Tier-1 Peering InitiativeDisruption #2: Large Tier-1 Providers create the

Next-Generation Peering Initiative

Common facilities are chosen, to which each major Tier-1 will acquire fiber rights

Allows excellent scalability while minimizing repeated cost

Independence, Fault Tolerance, and Provisioning time improve

Next-Generation Tier-1 Peering: Before N*(N-1)/2 Links in

each of 7 Cities Dependence on

RBOCs Slow Provisioning Mostly OC-3 and

OC-12 Poorly adapted to

use of Dark Fiber

Spr

UU

Gen

CW

ATT

L3

Q

Next-Generation Tier-1 Peering: After One-time Dark

Fiber provisioning to a single facility

Each Tier-1 maintains routers in the same room

Cheap, fast cross-connects for OC48 or higher peering

Spr

UU

Gen

CW

ATT

L3

Q

NexGen Peering Center

Peering Cost – Appearance of Metro Ethernet

Disruption #3: Cheap Metro Ethernet services connect to multiple peers over the same port using Point-to-Point and Multipoint VLANs

Metro-Ethernet Multipoint Private Peering

$-$100$200$300$400$500$600$700$800$900

10 25 65 160

400

1000

2500

6500

Bandwidth (Mbps)

Co

st

Transit

Private Lines

Metro Ethernet

Disruptions – What do they mean?

Disruption: Low-Cost Transit

Legacy Peering methods lose popularity – NAP attachments and Private Line Peering links are no longer saving money.

Some companies retain NAP presence just to meet contractual requirements, or for marketing reasons.

Increased pressure to find cheaper Peering methods.

Next-Gen Tier-1 Peering: Implications Large Tier-1s draw smaller Tier-1s to new

Centers for Peering

Largest Tier-2 and Content providers drawn to the Centers hoping to peer with Tier-1s

Smaller Content, Tier-2s, Enterprises and others are drawn to new Centers for competitive Transit and some Peering

What happens to all the other collocation?

Price Disruption: Metro-Ethernet Most major Collocation spaces in “NFL Cities”

are already Metro-Ethernet enabled

Pressure to replace individual Private Lines (DS-3, OC-3, OC-12) with Metro-Ether where possible.

Legacy NAP viability may depend on ability to accommodate Ethernet links Many are ATM-based and charge heavily for

collocation Some have adapted or may yet adapt

Metro Multipoint Private Peering: Before

Private Peering NAP Peering In-Building Peering if

you’re lucky.

Low efficiency High Cost Slow Provisioning

Old NAP

Private DS3s

ATM OC3

ME

YOU

HIM

Those Guys

(The situation for Smaller Tier-1, Tier-2, and Content)

Metro Multipoint Private Peering: After

Cross-Connect to an Ethernet port in your current facility

Multiple VLANs for Private and Public Peering, and Transit

Link to NexGen Peering Centers?

ME

YOU

Those Guys

(The situation for Smaller Tier-1, Tier-2, and Content)

HIM

NexGen Peering Center

Transit Provider

PublicPeeringVLAN

?

Efficient, Cheap, and Fast

Metro Multipoint Private Peering Addresses some limitations of Private Peering

Multiple Peer links on the same physical port Private, Public, and Transit share same ports

Many configurations and payment options Pay by port, by utilization, or by allocation P-to-P VLANs per peer, or semi-public

Multipoint VLANs Bandwidth allocated per port or per VLAN Technical options: Broadcasts, Switch Config,

Failover, etc.

Metro Multipoint Private Peering

Probably won’t scale to tens of Gbps May scale further once 10Gig-E is available Scaling is better addressed by Dark

Fiber/Lambda connections to the NexGen Peering Centers

Optimum scalability only reached if all local peers use the same Metro-Ether Provider (!)

FUD*

* Fear, Uncertainty, and Doubt

FUD – The Tier-1 Peering Initiative

Q. Is this Unfair? Non-Inclusive? Anti-Competitive?

A. No… it’s just common sense, and anyone can join in.

Q. Will this increase Transit prices? A. Competition should improve Transit pricing in

the Peering Centers, and have no effect outside them.

FUD – The Tier-1 Peering Initiative

Q. What about my old CoLo? A. Thanks to cheap metro, most large

Collocation centers are still viable, but you may want to get a foot in the door at the new peering centers.

Q. Are the Tier-1’s really committed to this? A. Yes… It’s the only viable long-term option,

and they don’t want to waste the investment they’ve already made.

FUD - Metro-Multipoint Private Peering Q. Early Entry is scary… why not wait? A. Very low commitment is needed; can start

below 10M

Q. Does the service have the robustness of Private Line?

A. All vendors use diversity wherever available. Most use same or similar configuration to the standard SONET/Lambda services we all rely on

FUD – Metro-Multipoint Peering, cont’d Q. What about the financial stability of the

vendors? Isn’t the cheap pricing based on gaining market share? We saw what happened in the last year… What about today’s vendors? (Applies to Metro-Ether and Peering-Collocation vendors)

A. Reduce your risk with short contract terms…Multiple vendors means you can switch if need be.

FUD – Metro-Multipoint Peering, cont’d Q. What technical gotchas should we be aware

of? A. Management overhead similar to old NAP,

and Broadcast and Congestion issues need to be resolved

Bottom line: Mitigate risk with short contract terms, and do your own financial analyses…

Caveat Emptor!

Points of Contact: Earthlink

Jeb Linton (Architecture/Peering Strategy) – jeblinton@corp.earthlink.net

Josh Fleishman (Peering Coordinator) – flyman2@corp.earthlink.net

Backup: What Happens Next?

Backup: What Happens Next

1. In “NFL Cities”, expect most large Internet companies (and many small ones) to migrate to the NexGen Peering Centers.

2. In all areas with Metro-Ethernet, expect off-net Collocation Centers to die off.

3. Expect ATM-based NAPs to continue losing popularity.

4. Metro-Ethernet will predominate for new P-to-P Private Peering links and links to the NexGen Peering Centers.

5. Metro-E will be used for Multipoint Peering. (Services already emerging from several vendors)

6. One Metro-E vendor will begin to predominate for Peering in each Metro area.

7. Peers will migrate from MM-PP to the Centers as they grow.

Backup: What Happens Next… the Big Picture Large Internet companies interested in significant Peering

will migrate toward the NexGen Peering Centers Mid-sized companies will use both:

NexGen Peering Centers, for Tier-1 and High-Speed peering Metro-Multipoint for smaller peers, to maximize savings

Small companies will: Start with a Metro Ether port for Transit, B-B, etc. Add peering VLANs on the same port for cost savings

Point-to-Point for B-B links and opportunistic private peering Metro-Multipoint for additional cost savings

Use Metro Ethernet as a bridge to the NexGen Peering Centers as they grow

Backup - Terminology (for convenience only) Tier-1 – Providers of Internet Access to

Autonomous Systems (a.k.a. “Leased Transit Service”, “Transit”)

Tier-2 – Providers of Internet Access to Individuals

NexGen Peering Centers – Collocation Centers chosen as common locations for Peering by the Largest Tier-1 Providers

Backup - Who wants to peer with whom? Largest Tier-1

Each Other Other Tier-1

Largest Tier-1, each other, large Tier-2, and some Content Large Tier-2

Content, all Tier-1, and each other Content

Everybody but each other Other (Who is Other?)

Academe, Enterprises, R&D, etc… the Usual Suspects

Backup: The “Peering Promiscuity Chart”

LargestTier-1

OtherTier-1

LargestTier-2

Content

Other

OtherTier-2

Other Similar

Other Similar

Other Similar

Other Similar

Content-HeavyTraffic Mostly

Outbound

Eyeball-HeavyTraffic Mostly

Inbound

Arrows point toward Targeted Peers

Backup – Metro Ethernet

OpticalRouter

DWDM

Switch

Dual FiberPaths

Backup – Point-to-Point Metro Ethernet

ME YOU

Backup – Multipoint Metro Ethernet

ME YOU

THEY THEM

Backup - Evaluating Available Methods of Peering

Questions to Ask: How much Peering traffic is locally available using this peering

method? How much does this Peering Method cost per Mb for the

expected traffic level? What does the company pay per Mb for Transit in this metro

area?

Things to consider: Should be half the cost of Transit (per Mb) or better Methods of Peering are complimentary, not exclusive