1
22 v1.0
IXP & Routing Tutorial– Internet eXchange Point (IXP)
W E B I N A R C O U R S E
Che-Hoo ChengInfrastructure & Development Director, APNIC
2019-10-21
3 v1.03
Disclaimer
• This talk includes my personal experience and observations from operating the IXP in HK before joining APNIC in Apr 2017o Plus a bit of my additional experience and observations from helping the development of a
few IXPs in the region
• Some of the points to be presented may NOT represent the viewpoints of APNIC
• Try not to name names, if at all possible• Try to be more interesting, and educational• There is no “One Size Fits All”
o Just to provide hints, not answers
• Cannot cover all scenarios here because of limited time
4 v1.04
How Does Internet Operates?
• Internet is a network of networks, composed of networks of ISPs and users
• User networks connect to ISPs
• Small ISPs connect to large ISPs
• Various networks (large or small) are interconnected with one another to form Internet
5 v1.05
Autonomous Systems
• A network on Internet is called Autonomous System (AS) which is represented by AS Number (ASN)o ASN is unique around the world
▸ APNIC is in charge of ASN assignment for AP region
o Used together with BGP (Border Gateway Protocol) for interconnections with multiplenetworks (or multi-homing)
o Networks having ASNs can be more independent, or portable▸ Together with portable IP addresses▸ Like what APNIC members are enjoying…
6 v1.06
Transit Provider(Upstream)
Global Internet
DownstreamCustomer
Customer routes only
Routes of the whole world
All customer routes
Ordinary Transit Model
7 v1.07
Transit in General
• Networks pay transit providers to get to the whole Interneto Can connect to multiple transit providers for resilience and portability
• A few very large ISPs act as transit providers for the whole world (the so-called tier-1 networks) which do not need to pay others to get full Internet connectivityo Other ISPs must be transit customers of those tier-1 networks directly or indirectly in
order to gain full connectivity
• Networks on Internet are trying to bypass transit providers as much as possibleo for lower cost and higher performance
8 v1.08
Transit Provider A(Upstream)
Transit Provider B(Upstream)
DownstreamCustomer
DownstreamCustomer
DownstreamCustomer
DownstreamCustomer
Routes of A and its customers
Routes of B andits customers
Ordinary Peering Model
9 v1.09
Peering in General
• ASes are interconnected/peered at Internet exchanges points (IXPs) or privately
• Interconnection/peering is among ISPs / data centres / content providers / cloud services providers which have different ASNs using BGP protocol
• For higher performance, lower latency and lower cost• Usually no settlement between peers and cost is sharedFor mutual benefits
• Local-to-local traffic do NOT need to route through overseas• Important to local Internet developmentLocal Peering
• BLPA (Bi-Lateral Peering Agreement)Between 2 ASes
• MLPA (Multi-Lateral Peering Agreement)Among > 2 ASes
10 v1.010
Private Peering
• A form of BLPA having dedicated point-to-point connection between 2 ASes
• Using cross-connect or local loop or IPL to interconnecto Cost is usually shared between 2 peers
• May have multiple connections between 2 ASes for resiliency
• Not quite cost-effectiveo Spare bandwidth cannot be used for other traffic
• Not very scalableo nC2 physical connections for n ASes to peer fully with one another
11 v1.011
What is an Internet eXchange Point (IXP)?
• An IXP is a shared physical network infrastructure over which various Autonomous
Systems can do easy peering with one another
One physical connection to IXP can be used for interconnections with multiple networks
More cost-effective and scalable
ASes to be served by IXP include Internet Gateways, Internet Service Providers (ISPs), Research & Education (R&E) Networks, Cloud Service Providers, Content Providers and Content Delivery Network (CDN) Service Providers
12 v1.012
Benefits of IXP
• One main objective of an IXP is to keep local traffic localo Important to local Internet development
• Helps bypass 3rd-party network infrastructure for easy interconnection and direct traffic exchange among participating networks
o Reduced cost – cheaper connectivityo Enhanced network performance – faster speedo Reduced latency – lower delay
• Helps encourage development of more local content and local applicationso Helps local data centre business and other businesses
• Everybody is benefitedo The gain for each may be different but all will gaino At the end, it is the most important that end users or consumers are benefited
• Often considered as Critical Internet Infrastructure locally, regionally or globally
13 v1.013
IXPs are Layer-2 Networks
• Switched Etherneto One physical connection for interconnections with multiple networkso Only routers are allowed to connect to the switching fabric directly
• IXP participants can do direct Bilateral Peering (BLPA) over the layer 2 infrastructure anytime
• With Route Server added to the layer 2 infrastructure, IXP participants can also do Multilateral Peering (MLPA) for easier interconnections among everybodyo Traffic exchange is not going through the route server but direct
• Those called themselves “IXes” but serving layer-3 services are mostly transit providers
14 v1.014
Value and Attractiveness of an IXP
• Proportional to the number of different networks (ASNs) connected and also the traffic volume
• Snowball effect after reaching critical masso The initial period usually is the hardest
▸ Most will take wait-and-see approach
o Gradually will have good mix of networks of different types▸ E.g. Eyeballs vs Content
15 v1.015
Evolution
• IXP development is an evolutionary process done step by step
• It can be improved over time, but picking the right initial neutral organisation / governance model and a neutral site at the start is important for future success
16 v1.016
Possible Steps for IXP Development
• Can be gradual, step by step
• Layer-2 network is the bare minimalo Can use private IP addresses if small amount of participants
• Public IP addresses nexto Legal entity issue
• Site resilience is IMPORTANT while equipment resilience is already includedo Has to have site resilience sooner or later
• Route server(s) with ASN followso RPKI consideration
• Other value added serviceso DNS: Root / TLDs / Recursive o Shared Content Caches?
17 v1.017
IP/ASN Resources for an IXP
• Considered as Critical Infrastructure under APNIC Policyo Using public IP addresses and ASN is recommendedo IPv4: /24o IPv6: /48o ASN: 1 (for route server)
• But IXP may need another network to provide transito Own servers such as network management & monitoringo DNS anycast servers: Authoritative or Cache/Resolving/Recursiveo Shared Content Caches for Participantso Usually small
18 v1.018
Neutral Location is Good Starting Point
• May choose one of the followings as starting point:o Universityo Technology Parko Carrier Neutral Data Center o Government Data Center
• Having multiple carrier options with easy access is important
• Should maintain neutrality continuously
• Expansion to multiple sites within the same metro area can be done gradually, coupled with growth
19 v1.019
Governance
• Multi-stakeholder bottom-up approach is the preferred approach for maximum acceptance of the community
• Government support is also important
• Be as inclusive as possible in order to provide maximum benefits to the whole community which it serves
• Should be fair and consistent to every participant
• Should be open and transparent as much as possible
20 v1.020
Government-led vs Industry-led
Subsidized vs Self-financed
Non-profit vs Commercial
Developed economies vs Developing economies
IXP Models
• No one single model which can suit all situations
• Relative Neutrality is important
21 v1.021
Commercial vs Non-Profit
• Commercial set-up is free to do anythingo No need to care about neutrality too mucho IXP is mostly a service to help other business
• Non-profit set-up tends to be more cautiouso Neutrality is more important, at least to the target participantso Tend to be more independento Tend to offer fewer services
22 v1.022
Advanced / Developed Economies
• IXPs are businesso Even for not-for-profit set-upo Less government involvement
• Multiple IXPso Keen competition
• But if they cannot keep intra-economy traffic local, someone needs to step upo Government? Industry group? Customer pressure?
23 v1.023
Developing Economies
• Some do not have any IXPs yet
• Local traffic does not stay localo A lose-lose situation for everybody
• IXPs can help Internet development a loto Better to be non-for-profit set-upo May need to start with subsidized modelo May not be a business at allo Help from government is mostly neededo Active participation of the biggest players is also very important
24 v1.024
Examples of Pacific Islands
• Far from any other places• External connectivity is very expensive
o More submarine cables are being built for them
• Small markets because of small population• Usually just a few ISPs but they may not be interconnected locally• Local traffic across ISPs usually routed through US or Australia• Local IXP is very much needed for helping Internet development• Observed immediate benefits on Day 1 of set-up of one Pacific Island IXP
o Much improved latency and high volume of traffic
• Small land-locked economies have more or less similar issues
25 v1.025
Politics Involved in Early IXP Development
• Usually larger ISPs like IXP less than smaller ISPs because smaller ISPs are mostly target customers of larger ISPs
• Larger ISPs refuse to connect to IXP making the value of IXP lower• There are multiple possible mitigation options for that but in any case, larger ISPs
need to collaborateo E.g. separating access networks from Internet gateway or transit network
• If hurting the goal of “Keeping Local Traffic Local”, then it is lose-lose to everybody• Government involvement may help or may hurt the case
o It depends on the relationship between the industry and the governmento Forcing large ISPs to do peering may not achieve the expected outcomes
• But having an IXP is NOT a magic wand to solve all the issueso But collaborative spirit is
26 v1.026
Government Funding for IXPs?
• Is it good or bad?
• More needed during infancy stage of IXP development
• But for long-term, it is probably better to have bottom-up industry-led governance for IXPo Align with bottom-up multi-stakeholder approacho Need to have a long-term sustainable financial model
27 v1.027
Which Models Can Sustain?
• IXP alone cannot make big money• Or IXP may just be a value added service Pure Business Model
• Government funding may or may not be more reliableSubsidized Model
• Most risky as sponsorship or support of volunteers is not guaranteed
Model relying on sponsorship and/or
volunteers
• Open Membership vs Closed Membership• Proper governance is important• Most neutral but still need to have good financial
model for long-term sustainability
Membership-based Model
28 v1.028
Geography
• IXP usually is not expanded beyond a metro area so as to avoid competing with IXP participants and to maintain neutrality
• Should start with the biggest city first and gradually set up separate infrastructure in other bigger cities one by one
29 v1.029
To Leverage the Position of & to Add Value to an IXP
• Domain Name Infrastructure: DNS infrastructure is very important to Internet operations so Root/TLD DNS server instance(s) should be connected directly to IXP for direct peering in order to benefit all participants for better DNS performance and resilience
• Shared Cache: Connecting cache servers of popular content to the IXP will help everyone save bandwidth, but the cost of the bandwidth for cache-fill has to be properly shared by the ISPs benefitedo Different cache service providers have different supported modelso Need to think about long-term sustainability
• NOTE: Transit for the above should NOT be used for providing usual transit service to IXP participants so as to maintain neutrality
30 v1.030
IXP Participants
• Unfortunately, a lot of IXP participants do not make the best use of the IXP(s) they have connected
• IXP Participants without enough knowledge and skills may disrupt the operations of IXP from time to time
• IXP operators need to do a lot of education or push to their participants
• So, IXP engineers would be busy and dedicated resources would be neededo Volunteering type of operations mode cannot sustain for too long
31 v1.031
Success Factors of the IXP in HK
• Helped gain trust from the participants especially the early ones• But there is no 100% neutrality…o Competition from another university
• After gaining critical mass, things are much easier• No need to do sales work at all
Neutrality
• In the first 10 years or so• Little hesitation for participants to connect• But cannot be free forever
Free Service Initially
• Earlier than the incumbent telco starting its ISP businesso They even asked for joining before they launched the
service• History cannot be repeated that easily…
Started Early
32 v1.032
And also…
• Leveraging telecom deregulation in HK
• Leveraging existing networks
• Passion & persistenceo And, there was incentive for doing it
• Adaptation to industry changeso E.g. opening up to unlicensed networks
• HK people have been enjoying fast local Internet connectivity since almost the beginning
33 v1.033
Long-Term Misunderstandings
• Used to mention ”>98% of traffic” a lot
• Government people and general public always think >98% of external traffic is going through the IXP in HKo How can that be possibly true?!o It is just wishful thinking of those people
• But the more accurate wordings should be:o The IXP in HK helps keep >98% of local traffic local
34 v1.034
Other Misunderstandings
• The IXP supports Bilateral Peering since the beginningo Although it did emphasise Multilateral Peering in the early days
• The IXP is NOT the only IXP in HK o There are in fact multiple IXPso The IXP is just the earliest and the biggesto The other IXPs together are not really small
▸ Perhaps 70:30 in terms of traffic volume
o But the IXP is the focus of people, most of the time
35 v1.035
Multilateral Peering is evil?
• Mandatory MLPA established initially was meant to be for HK routes only
• Mandatory MLPA for HK routes did help attract some overseas ISPs to connect and then gradually made the IXP become Regional IXPo Personally think this was probably the most successful MLPA case
• Mandatory MLPA for HK routes was gradually “unmentioned” because of large content / CDN providerso Not big transit providerso Definitely not related to any other IXPs set up in HK
• Mandatory MLPA is not the norm all around the world now…o Large providers will find ways to get around it
• Personally do not like stripping away the ASN of the route server from AS Path as it helps identify the routes learned from MLPA more easily
• Route servers at IXPs should provide more options to participants by using BGP community instead of just providing plain MLPA so that participant can have more control of routing even over route servers
36 v1.036
Vulnerabilities of IXPs
• Why can’t all router vendors have Proxy ARP disabled as default?• Cannot stop it totally because of possible human errors• Can only do regular monitoring by checking the ARP table• EVPN over VxLAN technology should be able to help but it is not a simple technology
Proxy ARP
• May happen when there is asymmetric routing seen from an IXP• Can be mitigated by sending proactive ARP check to all active addresses every hour or so• EVPN over VxLAN technology should be able to help but it is not a simple technology
Unknown Unicast Flooding
• Can cause trouble to multiple connections when there is big congestion on one porto Unknown to innocent participants which do not have any congestion
• Just be careful when choosing switch modelso Also avoid switch models with small buffer
Shared Buffer over Multiple Switch Ports
37 v1.037
Vulnerabilities of Data Centres?
• Locations are knowno Same for Landing Stationso Can easily be targets of physical attacks
• How can you better protect the fibre lead-ins and manholes which are outside of data centres?
• No such things as absolute security…o But let’s still do our best
38 v1.038
Visibility of Traffic?
• Support of layer-3 sFlow/NetFlow highly desirable for better visibility of traffic going through the IXPo It helps trouble-shooting and understanding of traffic profile/pattern a loto Having visibility of just layer-2 data is of less use
• But participants and general public would be concerned about the perceived surveillance or monitoringo Should do the best not to give data away to 3rd-partieso IXP needs to maintain trust with the participants
39 v1.039
Port Security Is Important
• The IXP in HK allows just one MAC address per port (physical or virtual)o Strictly one IPv4 address, one IPv6 address and one MAC address per port (physical
or virtual)o Static MAC address for full controlo “Violation Restrict” instead of “Violation Shutdown”
• Minimum protection to the layer-2 broadcast domain
• A few IXPs allow more MAC address per port but still a small number
• Must also do Ether-type filtering and broadcast/multicast traffic filtering/rate-limiting
40 v1.040
Remote Layer-2 Connections to IXP?
• More and more common nowadayso Some even from >1,000km away
• Using fibre-only connection is much easier, with much fewer issues o ZX/ZR/ZR4 are up to 70-80km
• Clear-Channel remote layer-2 circuits with full transparency are rareo Unless you are willing to pay moreo Wasting a lot of effort to do trouble-shooting with carriers
• But IXPs cannot afford to not support themo As they want to have more business, sometimes through resellerso Unless their main business is data centre business
41 v1.041
Scalability Issue
• IXPs were not supposed to have any packet loss in its infrastructure o And with very low latency too
• Become an issue when IXP grow beyond one switcho Due to not enough ports or expanding to multiple sites
• Inter-switch links are the risko Over-subscription or not?
• Spine-and-leaf architecture helps a bit but not for all caseso Need to determine how much bandwidth from leaf to spine anywayo Still not ideal if there are adjacent leaf switches at one site
▸ All traffic among 2 adjacent leaf switches has to go to the spine first?
42 v1.042
SpineSwitch
SpineSwitch
LeafSwitch
LeafSwitch
LeafSwitch
n x 100GE/10GEInter-Switch
Links
n x 100GE/10GEInter-Switch
Links
ISP ISP ISP ISP ISP ISP ISP
SpineSwitch
SpineSwitch
LeafSwitch
LeafSwitch
100GE/10GE/GELinks
100GE/10GE/GELinks
LeafSwitch
Spine-and-Leaf Architecture
43 v1.043
MRTG of Aggregate Traffic
• It is less sensitive of courseo More an indication of the importance and the growth of an IXP
o Should not neglect the huge difference between showing 5-min data and 1-min data
o Should not neglect what traffic data is included – just the main broadcast domain or what?
Usually incoming = outgoing
If incoming > outgoing Congestion at at least one port• May be DDoS attacks
If outgoing > incoming May have Unknown Unicast Flooding
If sudden drop of large traffic volume
May have Proxy ARP problem• Usually happens when change of router / router software /
router config
44 v1.044
Other Observations from MRTG
• Situation in HKo Holiday Effecto Soccer Games Effecto Typhoon Effect
• Difference of Culture / Practiceso E.g. HK vs Japan
45 v1.045
IXPs and Data Centres
• They are natural partners
• Common situation in advanced metro citieso Multiple IXPs in one Data Centreo One IXP in multiple Data Centres
▸ Should be the same layer-2 broadcast domain▸ Circuit cost is a burden
o Healthy competition would be good▸ Customers have choices▸ Also for better resilience
46 v1.046
IXP across Multiple Cities / Economies
• May not be good for maintaining neutralityo Considered as competing with participants which have presence in the same
locations
• Commercial IXPs can take this business risk especially if this may help their other business
• But not so good for non-profit IXPs targeting all kinds of networks or providerso Those that see competition may not join and then it may affect the goal of “keeping
local traffic local”
47 v1.047
Interconnections of 2 or More IXPs
• What are the purposes of doing this?
• Not considered a good idea at Layer-2, especially if across cities or countries/economies
• Even at Layer-3, still need to be mindful of whether it affects the original purposes of each IXP involved
48 v1.048
Shared Content Caches Offered by IXP?
• A lot of misunderstandings about the use of cacheso Caches may not always help save cost or improve performance
• A lot of local IXPs want to provide shared caches for their participants to increase their valueo Cost recovery and cost sharing / accounting are major issues to them though
• Content / CDN providers are still sceptical about this modelo They still mostly look at cache efficiency (cache hit ratio) and traffic volume for
justifications
• But it is still good to consider it
49 v1.049
Myth of Neutrality
• There is NO absolute neutrality• Different organisation has different perspective of neutrality
o A university?o A carrier-neutral data centre?o An IXP?o A government department?o A membership-based organisation?
• We can only be “very neutral” for a defined group of companies or organisations, but not for all…
• But maintaining higher relative neutrality is still better for IXPs
50 v1.050
Competition / Comparison among IXPs
• Especially when they are in the same metro area• Traffic Volume
o Apple-to-apple?o 5-min vs 1-min?o One broadcast domain vs multiple broadcast domains?o PNNI traffic added?o 1Tbps Club…
• Number of ASNs connected• Number of Ports connected• Amount of total connected customer bandwidth• #1 IXP in Asia???
51 v1.051
Threats to IXP Business
• /Mbps pricing of IP transit bandwidth is dropping continuouslyo Partly because of price drop of submarine cable capacityo /Mbps pricing of IXPs cannot be dropped as fast because of different cost base
▸ Equipment cost doesn’t drop a lot especially for high-end switches▸ Local loop cost involved for interconnecting multiple sites does not drop as fast
• More and more content caches are being set up inside the access networkso But bandwidth is still needed for cache-fill
• Private peering will take away traffic from IXPso If traffic volume warrants between any two parties
52 v1.052
The Way Forward for IXP Business
• It is tough business if you only do IXP businesso Fighting for survival
• Adding Value Added Services may helpo PNNI over VLANs, GRX, Cloud Exchange, GXP and etc
• Partnershipo Partnering with multiple Data Centreso Partnering with multiple local loop providerso Recruit resellers – local & overseas/global
• Expand overseaso A few European IXPs are doing this
• Merger and Acquisitiono Even non-profit set-up should get ready for this
53 v1.053
IXP Development Work of APNIC
• APNIC strongly believes IXPs help Internet developmento That is why we support APIX and related activitieso After all, IXPs serve and benefit APNIC memberso In fact, IXPs need IP addresses and ASNs and so are APNIC members themselves
• Do more on helping those developing economieso Especially those which do not have any IXP yeto Or those which their only IXP is not functioning
• Training and Technical Assistance work primarilyo Not just for IXP operators but also for IXP participantso Also help talk to major stakeholders to convince them of the benefits of having a local IXP while maintaining neutralityo May need help of Community Trainers and Consultants from time to time
• Having been supporting IXP development in Fiji, PNG, Vanuatu, Mongolia, Bhutan, Myanmar, some cities in India and others
54 v1.054
IXP Development Package of APNIC
• Providing Training & Technical Assistance is the minimum • Will tailor-make support according to individual needs• Other possible support items (on a case-by-case basis):
o Ethernet switcho Root Server anycast instanceo Route Servero ROV & IPv6 deployment supporto IXP Managero RIPE Atlas Anchorso CSIRT Establishmento Honeypot of Honeynet Project for Analysiso BGP Route Collection for Analysis
• APIX Membership is recommended to all IXPs
55 v1.055
Route Origin Validation (ROV) at IXP – via Route Server and/or Shared Validator
Validated cache
Validator
RPKI-to-Router (RTR)
Routes
Tagged/filteredroutes
RS
56 v1.056
Other Help & Support by APNIC
• APNIC also provides help & support to:o Peering Asiao Peering Forums hosted by not-for-profit IXPso NOGs (which IXPs usually support)
• APNIC also sponsors:o PeeringDBo IXP-DBo IXP Manager
57 v1.057
Final Remarks
• IXPs will continue to play a key role for easier interconnections among networkso Especially for developing economieso But IXP is NOT a magic wand to solve all the issues
▸ Collaborative spirit is
• Need to find a suitable model for long-term sustainability
• Relative neutrality is importanto So better to maintain it as much as possible
• After all, “Keeping Local Traffic Local” is the most important thing
58 v1.058
Thank You!END OF SESSIONThank You!
END OF SESSION
59