Sofía Silva Berenguersofia @ lacnic.net

Paramaribo - Surinam

Internet Exchange PointsWorkshop

AGENDA• How the Internet Works• Intro to BGP• IPv4 Exhaustion and IPv6 Deployment• Internet Exchange Points• How to request Internet Resources• Advanced topics

– Route Hijacking– Leaks– Attacks against the path– Well known incidents– Securing the Routing System

HOW THE INTERNET WORKS

Interconnection of Networks

http://prezi.com/agzxt0exlfpk/network-interconnection/

Internet Routing

ASN 6057 announces

200.40.0.0/16

The prefix 200.40.0.0/16 is propagated with BGP to the

InternetASN 8158 receives

200.40.0.0/16 Atributos: 200.40.0.0/16 AS_PATH ASN1 ASN3 ASN6057

Transit and Peering• Transit

– Traffic and prefixes originating from one AS are carried across an intermediate AS to reach their destination AS

– Usually for a fee• Peering

– Private interconnect between two ASNs– Usually for no fee

Transit and Peering

ASN 64511

ASN 65536ASN 65537

Peering

ASN 65538

Transit

Peering in an Internet Exchange Point (IXP)

• Internet Exchange Point – Common interconnect location where several

ASNs exchange routing information and traffic

ASN 65536

ASN 65537

ASN 65538

ASN 65539

IP address, where they come from?

Sometimes the distribution is done through National Internet Registries (NIRs)

Regional Internet Registries (RIRs) distribute IPv4, IPv6 and Autonomous System Numbers

Standards

Central Registry

Distribution

Allocations and Assignments

End user

*

*

Distribution

Regional Internet Registries

INTRO TO BGP

Border Gateway Protocol• A Routing Protocol used to exchange routing

information between different networks• Exterior gateway protocol• Described in RFC4271

– RFC4276 gives an implementation report on BGP– RFC4277 describes operational experiences using

BGP• Works on TCP port 179

More about BGP• Learns multiple paths via internal and external

BGP speakers – Initial exchange of entire table• Incremental Updates

– Picks THE bestpath and installs it in the IP forwarding table – Policies applied by influencing the bestpath selection

• Keepalive messages exchanged• Many options for policy enforcement• Classless Inter Domain Routing (CIDR) • Widely used for Internet backbone

Neighbors

• BGP speakers– Internal (iBGP) if they are in the same ASN– External (eBGP) if they are in different ASN

ASN 65536

ASN 65538

eBGPiBGP

Where to use BGP: Stub Network

ASN 65536, Transit Provider

ASN 65538, Customer

• Only one exit for customer

• Not really need to add BGP

Multihomed Network

ASN 65536

ASN 65537

ASN 65538ASN 65539

Transit Providers

Peering in IXP

• Different situations possible• Multiple links to same ISP• Secondary for only backup• Load share between

primary and secondary • Selectively use different

ISPs• Peering at IXP

INTERNET EXCHANGE POINTS

Why peer?• Consider a region with one ISP

– It provides internet connectivity to it’s customers– It has one or two international connections

• Internet grows, another ISP sets up in competition– They provide internet connectivity to their

customers– They have one or two international connections

• How does traffic from customer of one ISP get to customer of the other ISP?– Via the international connections

Why peer? (Cont.)

ASN 65536ASN 65538

Internet

Why peer? (Cont.)• Yes, international connections…

– If satellite, RTT is around 550ms per hop– So local traffic takes over 1s round trip

• International bandwidth…– Is much more expensive than domestic bandwidth– Becomes congested with local traffic

• Wastes money, harms performance

Why peer? (Cont.)• Solution:

– Two competing ISPs peer with each other• Result:

– Both save money– Local traffic stays local– Better network performance– More international bandwidth for international

traffic

Why peer? (Cont.)

ASN 65536ASN 65538

Internet

Why peer? (Cont.)• A third ISP enters the equation

– Becomes a significant player in the region– Local and international traffic goes over their

international connections• They agree to peer with the two other ISPs

– To save money– To keep local traffic local– To improve network performance

Why peer? (Cont.)• Peering means that the three ISPs have to buy

circuits between each other– Works for three ISPs, but adding a fourth or a fifth

means this does not scale• Solution:

– Internet Exchange Point

Why peer? – Non-financial Motivations• Low latency• Control over routing• Redundancy• Aggregation benefits w/peering and Transit at

IXP• ISP relationships – be one of the cool kids• Marketing benefits• Network reliability

Internet Exchange Point• Every participant has to buy just one whole

circuit– From their premises to the IXP

• Rather than N-1 half circuits to connect to the N-1 other ISPs– 5 ISPs have to buy 4 half circuits = 2 whole circuits

-> already twice the cost of the IXP connection

Simple Topology

• Layer 2 fabric• N^N BGP relations

IXP Design• Each ISP participating in the IXP brings a

router to the IXP location• Router needs:

– One Ethernet port to connect to IXP switch– One WAN port to connect to the WAN media

leading back to the ISP backbone– To be able to run BGP

IXP Design (Cont.)• IXP switch located in one equipment rack

dedicated to IXP– Also includes other IXP operational equipment

(Management network, TLD DNS, Routing Registry, Looking Glass, etc.)

– Optional: Second switch for redundancy• Routers from participant ISPs located in

neighbouring/adjacent rack(s)• Copper (UTP) connections made for 10Mbps,

100Mbps or 1Gbps connections• Fibre used for 10Gbs and higher speeds

Peering at an IXP• Each participant need to run BGP

– They need their own AS number– Public ASN, NOT private ASN

• Each participant configures external BGP with the other participants in the IXP– Peering with all participantsOr– Peering with a subset of participants

IXP - Routing• ISP border routers at the IXP generally should

NOT be configured with a default route or carry the full Internet routing table– Carrying default or full table means that this

router and the ISP network is open to abuse by non-peering IXP members

• ISP border routers at the IXP should not be configured to carry the IXP LAN network within the IGP or iBGP– Set BGP next-hop to local router (Cisco IOS next-

hop-self)

IP Address Space• Some IXPs use private addresses for the IXP

LAN– Public address space means the IXP network can

be leaked to the Internet, which could be undesirable

– Filtering RFC1918 address space by ISPs is Best Practice; this avoids leakage

• Some IXPs use public addresses for the IXP LAN– Address space is available from LACNIC– IXP terms of participation usually forbid carrying the

IXP LAN addressing in the ISP backbone

Hardware• The IXP Core is an Ethernet Switch

(Mandatory)– Therefore invest in the best and most expandable

equipment that its financial circumstances allow– Having 2 switches is good for redundancy if the

funds can allow• Route Server (Optional)

– Provides ease of configuration for new members– Direct peering between the IXP members can be

implemented in the absence of the Route Server

Hardware (Cont.)• Other optional equipment

– Web Server (website, monitoring, etc.)– Mail Server (email, mailing list, etc.)– Transit Router (to provide Internet access to the

IXP website, email and staff Internet access)– Route Collector (Looking glass which assists IXP

members with troubleshooting. It can also be used to collect routes for statistics measurements)

Hardware - Suggestions• Try not to mix port speeds

– If 10Mbps and 100Mbps connections available, terminate on different switches

• Insist that IXP participants bring their own router– Moves buffering problem off the IXP– Ensures integrity of the IXP– Security is responsibility of the ISP, not the IXP

Location• The location of the IXP is very important.• The IXP location should be neutral and low cost.• In considering the IXP location the following factors

should be considered:– Space– Environment Control– Security– Power– Access to terrestrial Infrastructure– Cabling– Support

Recommendations and Best Practices• Only announce your aggregates and your

customer aggregates at IXPs • Only accept the aggregates which your peer is

entitled to originate • Never carry a default route on an IXP (or

private) peering router• Failing to do so leads to route-hijacks and

leaks

General Info about IXPs

Source:https://prefix.pch.net/applications/ixpdir/summary/

.

.

....

General Info about IXPs

Source: https://prefix.pch.net/applications/ixpdir/?show_active_only=0&sort=traffic&order=desc

.

.

.

General Info about IXPs

Source: https://prefix.pch.net/applications/ixpdir/summary/ipv6/

HOW TO REQUEST INTERNET RESOURCES

• Go to https://solicitudes.lacnic.net/• Or fill out a form and send it in the body of a

message to hostmaster@lacnic.net– You can find templates at:http://lacnic.net/templates/

• Once the online request or the form has been processed by the system, the requestor will receive a confirmation email with a ticket number.

• After that the hostmasters will analyze the request.• If the request is approved, it may be necessary to pay

a fee and to sign the Registration Service Agreement.

Who can request resources?

• The person allowed to request resources for an organization is the Administrative POC.

• To request resources through the new Requests System you will have to log in using the Administrative POC handle.

Requesting an ASN

• In order to qualify for an ASN allocation the organization should have:– A unique routing policy, meaning a policy that differs from that applied

by the upstream provider.– Or, a network with more than one independent connection to the

Internet. (Multi-homed site)• From January 1, 2007 to December 31, 2010 Lacnic assigned ASN

of 16 and 32bits upon request. However, since January 1, 2011 Lacnic stopped making distinctions between the assignment of 16- and 32-bit Autonomous Systems Numbers (ASNs) and will only assign ASNs from a general 32-bit pool. This change will be introduced to comply with the Global Policy "Internet Assigned Numbers Authority (IANA) Policy for Allocation of ASN Blocks to Regional Internet Registries" adopted in September 2010.

Micro-assignments to Critical Infrastructure

• Micro-assignment -> prefixes between /24 and /20.• For projects and network infrastructure that are key or critical for the region, such

as IXPs (Internet Exchange Points), NAPs (Network Access Points), RIRs, ccTLDs, among others.

• IXPs or NAPs must meet the following requirements:– Duly document the following aspects:

• Prove by means of their bylaws their IXP or NAP capacity. The organization shall have at least three members and an open policy for the association of new members.

• Submit a diagram of the organization's network structure.• Document the numbering plan to be implemented.

– Provide a utilization plan for the following three and six months.– If the applicant does not already have an IPv6 block assigned by LACNIC, simultaneously

request an IPv6 block in accordance with the corresponding applicable policy.• The rest of the applications shall be studied based on the analysis of the

documentation justifying the critical and/or key aspects of the project.• Organizations receiving micro-assignments shall not sub-assign these IPv4

addresses.

Requesting an IPv4 block for ISPs• To qualify for the allocation of a /22 block the org

must:– Prove usage or immediate necessity of a /24– Submit a detailed one-year usage plan for a /23– Agree to renumber from previously allocated space and

return those IP addresses to their ISPs within 12 months– If the applicant does not already have an IPv6 block

assigned by LACNIC, simultaneously request an IPv6 block in accordance with the corresponding applicable policy.

• For a larger block additional requirements apply

Requesting an IPv6 block for ISPs• To qualify for an initial allocation of a /32 block the

organization should:– Be a LIR (Local Internet Registry), which means being an

organization that assigns address spaces for its network services customers

– Not be an end site (end user)– Document a detailed plan for the services and IPv6 connectivity

to be offered to other organizations (clients)– Announce the allocated block in the Internet inter-domain

routing system, with the minimum possible level of disaggregation to the one that is publishing the IP blocks, within a period no longer than 12 months.

– Offer IPv6 services to clients physically located within the region covered by LACNIC within a period not longer than 24 months

More info

• Policy Manual– http://www.lacnic.net/web/lacnic/manual

• Registration Services– http://www.lacnic.net/web/lacnic/servicios-registro

ADVANCED TOPICS

Route Hijacking• This occurs when a participant in the Internet

Routing announces a prefix for which it has no authority

• Malicious or by operational errors• More know cases:

– Pakistan Telecom vs. You Tube (2008)– China Telecom (2010)– Google in Eastern Europe (various AS, 2010)– Latin American cases (beginning 2011)

Route Hijacking

AS 15358 announces

200.40.235.0/24ASN 8158 receives

200.40.0.0/16 y 200.40.235.0/24 200.40.0.0/16 AS_PATH ASN1 ASN3 ASN6057

200.40.235.0/24 AS_PATH ASN1 ASN15358

AS 6057 announces 200.40/16

ASN 8158 receives

200.40.0.0/16

Leaks• There is not a standard definition of leaks• But it happens when an ASN “leaks” non-

customer or self-originated routes to other peers.

• The effects is to give transit to those networks for the peers of the ASN

Route Leaks

ASN 64511

ASN 65536ASN 65537

Transit

Peering2001:db8::/40

2001:db8:100:/40

2001:db8::/40 655362001:db8:100::/40 65537How this

should work without leaks

2001:db8::/402001:db8:1/48

2001:db8::/40 2001:db8:100:/40Traffic to whole 2001:db8::/40 goes this way

Route Leaks

ASN 64511

ASN 65536ASN 65537

Transit

Peering

2001:db8::/40

2001:db8:100:/40

2001:db8::/40 655362001:db8:100::/40 655372001:db8:1::/48 65536 65537

Now a Route Leak

2001:db8::/402001:db8:1/48

2001:db8::/40 2001:db8:100:/402001:db8::/402001:db8:1::/48

Traffic to 2001:db8:1::/48 goes this way

Attacks against the path• AS Insertion:

– A router might insert one or more ASNs, other than its own ASN, into an update message

• False (Route) Origination with valid ASN: – A router might originate a route for a prefix using

an ASN not authorized to originate routes for that prefix.

Attacks against the path

Fake AS 6057 announces

200.40.235.0/24ASN 8158 receives

200.40.0.0/16 y 200.40.235.0/24 200.40.0.0/16 AS_PATH ASN1 ASN3 ASN6057

200.40.235.0/24 AS_PATH ASN1 ASN15358

AS 6057 announces 200.40/16

ASN 8158 receives

200.40.0.0/16

ASN 6057

WELL KNOW INCIDENTS

Pakistan Telecom vs. Youtube• On Sunday, 24 February 2008, Pakistan Telecom

(AS17557) started an unauthorized announcement of the prefix 208.65.153.0/24 (Youtube)

• One of Pakistan Telecom's upstream providers, PCCW Global (AS3491) forwarded this announcement to the rest of the Internet, which resulted in the hijacking of YouTube traffic on a global scale.

• Reason: “Fat fingers”• Video de RIPE NCC

– http://www.youtube.com/watch?v=IzLPKuAOe50

Moratel vs Google• Reported by Cloudflare on November 06, 2012• Google's services experienced a limited

outage for about 27 minutes over some portions of the Internet.

• Moratel (23947) was “leaking” Google one route and packets were going through Indonesia

• Reason: “Fat fingers”

ASN and Prefix Hijack• On 2011 one large European ISP complain that

one of their prefixes was being announced by a Mexican ISP

• The Mexican ISP review their network but could not found the problem

• Later it appears that a Brazilian ISP was using the Mexican ISP’s ASN to announce the European prefix

• Reason: Poor BGP knowledge

SECURING THE ROUTING SYSTEM

Recall how Internet Resources are managed

Recall how Internet Resources are managed

• Each RIR is an authoritative source of information about the relation “user” <-> “resource”

• Each RIR operates its registration data base• Members and RIRs sign Service Agreements between them

Who has the "right" to use resources?• When an ISP obtains resources from its RIR (IPv6/IPv4/ASN):

– The ISP has to notify its upstream ASs which prefixes are going to be announced via BGP

– This is usually done via e-mail, web forms or by updating an IRR (Internet Routing Registry)

• Upstreams verify (or at least they should) the right of use for the announced resources– RIR WHOIS Text-based and not really suitable for automatic usage– IRR WHOIS Non-signed information, little additional tools

provided for verification of usage rights except for names, phone numbers and email POCs

• This verification process is sometimes not as thorough as it should be

What is RPKI?

• RPKI (Resource Public Key Infrastructure) allows the validation of an organization right to use of a certain resource (IPv4, IPv6, ASN)

• RPKI combines the hierarchy of the Internet resource assignment model through RIRs with the use of digital certificates based on standard X.509

• RPKI is standardized in the IETF through the SIDR WG. It has produced RFCs 6480 – 6492

RPKI• All RPKI signed objects are listed in public repositories• After verification, these objects can be used to

configure filtering in routers• Validation Process

– Signed objects have references to the certificate used to sign them

– Each certificate has a pointer to an upper level certificate– The resources listed in a certificate MUST be valid subsets

of the resources listed in its parent's certificate– In this way a trust chain can be traced to a "trust anchor"

both cryptographically as well as in CIDR terms

X.509 v3 certificates with RFC 3779 extensions

• X.509 Digital Certificates– Subject, validity period, public key

and other fields• With extensions:

– RFC 3779 defines extensions that allow the representation of Internet resources as certificate fields

• List of IPv4, IPv6 and ASNs assigned to an organization

• Implemented in OpenSSL 1.0c onwards

Signature Algorithm

Serial Number

Version

Issuer

Subject

Subject Public Key

Extensions

Addr: 10.10.10.0Asid: 65535

Subject Information Authority (SIA)

Authority Information Access (AIA)

ROAs• Using Certificates we can create objects

describing the origin of a prefix• ROAs: Routing Origin Authorization

– ROAs contain data on the allowed origin-as for a set of prefixes

– ROAs are signed using the certificates generated by the RPKI

– Signed ROAs are copied to the repository

Prefix MaxLen Origin AS Valid Since Valid Until

200.40.0.0/17

20 6057 2013-01-02 2013-12-31

200.3.12.0/22

24 28000 2013-01-02 2014-12-31

ROAs (ii)

• A simplified ROA contains the following information:

• These ROAs states that:– "The prefix 200.40.0.0/17 will be originated by ASN 6057

and could be de-aggregated up to /20" "This statement is valid starting on Jan 2, 2013 until Dec 31, 2013"

• Other ROA content– ROAs contain cryptographic material that allows

validation of the ROAs content

ROAs (iii) - Validation• ROAs validation process includes:

– Criptographic validation of End Entity certificates (EE) that are included in each ROA

– CIDR validation of resources listed in the EE against the resources listed in the issuing certificate

– Verification that prefixes listed in the route origin attestations are included in the prefixes listed in the EE certificates of each ROA

Creation of ROAs

Creation of ROAs

Origin Validation• Routers build a database with the information

they receive from the caches• This table contains

– Prefix, Min length, Max length, Origin-AS• By applying a set of rules a validity status is

assigned to each UPDATE prefix• Network operators can use “validity” attribute

to construct routing policies

IP prefix/[min_len – max_len] Origin AS

172.16.0.0 / [16-20] 10

200.0.0.0/[8-21] 20

• If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"

• If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"

• If the origin-AS does NOT match -> "invalid"

UPDATE 200.0.0.0/9 ORIGIN-AS 20

VALID

Origin Validation

IP prefix/[min_len – max_len] Origin AS

172.16.0.0 / [16-20] 10

200.0.0.0/[8-21] 20

• If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"

• If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"

• If the origin-AS does NOT match -> "invalid"

UPDATE 200.0.0.0/22 ORIGIN-AS 20

INVALID

Origin Validation

IP prefix/[min_len – max_len] Origin AS

172.16.0.0 / [16-20] 10

200.0.0.0/[8-21] 20

• If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"

• If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"

• If the origin-AS does NOT match -> "invalid"

UPDATE 200.0.0.0/9 ORIGIN-AS 66

INVALID

Origin Validation

IP prefix/[min_len – max_len] Origin AS

172.16.0.0 / [16-20] 10

200.0.0.0/[8-21] 20

• If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"

• If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"

• If the origin-AS does NOT match -> "invalid"

UPDATE 189.0.0.0/9 ORIGIN-AS 66

NOT FOUND

Origin Validation

Routing Policies with Origin-validation• Using the BGP validation attribute network

operators can construct routing policies• For example:

– Assign higher preference to routes with status “valid” than routes with status “unknown”

– Drop routes with status of “invalid”• Very important, RPKI is a source of data,

operators are free to use it as it suits them better

RPKI in action

Cache

RPKI Management System

Repository

Caches retrieve and cryptographically validates certificates and ROAs from repositories

Caches feeds routers using RTR protocol with ROA information

Routers validate updates from other BGP peers

RPKI Operation Modes• “Hosted” mode

– LACNIC issues certificates and keeps public and private keys in its systems

• Certificates are issued at the request of member organizations

– LACNIC provides a web interface for management• “Delegated” mode

– Each organization has it’s own certificate, signed by LACNIC’s CA

– The organization sends signing requests to LACNIC, who returns them signed

• “Up-down” protocol

RPKI today• 5 CAs and repositories working (RIRs) in hosted

and delegated mode (just APNIC)• Validation, CA and origin validation working

software and devices• Some other tools built around (bgpmon, LACNIC

Labs, RIPE Labs)• ~ 11,700 routes signed (~ 8.9% 1,226 invalid or

hijacks) • Origin Validation available in Cisco, Juniper,

Quagga, BIRD

LACNIC’s RPKI Structure

Self-signed RTA

Signing chain

LACNIC’s RPKI Structure (ii)• CAs

– Entity that issues certificates (bit CA=1)• ISPs can use the certificate to sign it’s clients’

certificates

• Certificates Repository– Repository of certificates, CRLs and manifests– Accesible through “rsync”

• Management Interface– User web interface for those who prefer “hosted”

mode

RPKI CA Services• Children certificates issuance when the

registration database is updated or on user demand

• Children certificates revocation in a centralized manner or on user demand

• Periodic issuance of CRL for CA certificate• CA Certificate and children certificates

publication in a public repository (rsync)

Conclusions• The routing system is one of the core

operations of the Internet• Still is vulnerable to attacks and bad configs• Some work has been done (RPKI, Origin

Validation)• We need to continue our work

– Protocol specification– Deployment (Filtering, RPKI, Origin validation)

Links / References• LACNIC’s RPKI System

– http://rpki.lacnic.net• LACNIC’s RPKI Repository

– rsync://repository.lacnic.net/rpki/• To see the repository

– rsync --list-only rsync://repository.lacnic.net/rpki/lacnic/

• RPKI Statistics– http://www.labs.lacnic.net/~rpki

Questions?

Comments?

twitter.com/LACNIC facebook.com/ LACNIC youtube.com/user/lacnicstaff gplusme.at/LACNIC

CASA DE INTERNET DE LATINOAMÉRICA Y EL CARIBE

Sofía Silva Berenguersofia @ lacnic.net

Thank you!