ISP Network Design - Network Startup Resource Center … · ISP Network Design ISP/IXP Workshops...

1

1© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops

ISP Network DesignISP/IXP WorkshopsISP/IXP Workshops


ISP Network Design

• PoP Topologies and Design• Backbone Design

• ISP Systems Design

• Addressing• Routing Protocols

• Security

• Out of Band Management• Operational Considerations


Point of Presence Topologies


PoP Topologies

• Core routers – high speed trunk connections• Distribution routers and Access routers – high

port density

• Border routers – connections to other providers• Service routers – hosting and servers• Some functions might be handled by a single

router


PoP Design

• Modular Design• Aggregation Services separated according to

connection speedcustomer servicecontention ratiosecurity considerations


Modular PoP Design

Backbone linkto another PoP

Backbone linkto another PoP

Nx64 customeraggregation layer

Nx64 leased line circuit deliveryChannelised T1/E1 circuits

Hosted ServicesISP Services(DNS, Mail, News,

FTP, WWW)

NetworkOperations

Centre

ConsumerDIAL Access

Other ISPsWeb Cache

NetworkCore

Consumer cable, xDSL and

wireless Access

NxT1/E1 customeraggregation layer

T1/E1 leased line circuit deliveryChannelised T3/E3 circuits

2


Modular Routing Protocol Design

• Modular IGP implementationIGP “area” per moduleaggregation/summarisation where possible into the core

• Modular iBGP implementationBGP route reflector cluster per modulecore routers are route-reflectorsclients peer with core only


Point of Presence Design


PoP Modules

• Low Speed customer connectionsPSTN/ISDN dialuplow bandwidth needslow revenue, large numbers

• Medium Speed customer connections56/64K to sub-T1/E1 speedslow bandwidth needsmedium revenue, medium numbers


PoP Modules

• High Speed customer connectionsE1++ speedsmedium bandwidth needshigh revenue, low numbers

• Broad Band customer connectionsxDSL, Cable and Wirelesshigh bandwidth needslow revenue, large numbers


PoP Modules

• PoP CoreTwo dedicated routersHigh Speed interconnectBackbone Links ONLYDo not touch them!

• Border Networkdedicated border router to other ISPsthe ISP’s “front” doortransparent web caching


PoP Modules

• ISP ServicesDNS (cache, secondary)News, Mail (POP3, Relay)WWW (server, proxy, cache)

• Hosted ServicesVirtual Web, WWW (server, proxy, cache)Information/Content ServicesElectronic Commerce

3


PoP Modules

• Network Operations Centreprimary and backup locationsnetwork monitoring

statistics and log gatheringdirect but secure access

• Out of Band Management NetworkThe ISP Network “Safety Belt”


Low Speed Access Module

To Core Routers

Primary Rate T1/E1

PSTN lines tomodem bank

PSTN lines tobuilt-in modems

AS5300

AS2511

2600/3600

TACACS+/Radiusproxy, DNS resolver,

Content

Web Cache

Access NetworkGateway Routers


Medium Speed Access Module

To Core Routers

Channelised T1/E1

64K and nx64K circuits

Mixture of channelisedT1/E1, 56/64K and

nx64K circuits

3800/7206/7600


High Speed Access Module

To Core Routers

Channelised T3/E3

T1 and E1 circuits

Mixture of channelisedT3/E3 and T1/E1 circuits

7200/7600


Broad Band Access Module

To Core Routers

Telephone Network

The cable system

6400

SSG, DHCP, TACACS+or Radius Servers/Proxies,

DNS resolver, Content

Web Cache

Access NetworkGateway Routers

uBR7246

61xx

IP, ATM


ISP Services Module

DNScache

DNSsecondary POP3 Mail

Relay NEWS

To core routers

WWWcache

Service NetworkGateway Routers

4


Hosted Services Module

Customer 7Customer 3Customer 4

Customer 5Customer 6

To core routers

Hosted NetworkGateway Routers

Customer 2Customer 1


Border Module

To core routers

NetworkBorder Routers

To local IXP -NB - no default route +

local AS routing table only

ISP1 ISP2


NOC Module

Primary DNS

To core routers

Hosted NetworkGateway Routers

SYSLOGserver

TACACS+server

Network Operations Centre Staff

Out of BandManagement Network

2620/32async

NetFlowAnalyser

Firewall

Billing, Database and Accounting

Systems

Corporate LAN

Critical ServicesModule


Out of Band Network

Out of BandManagement Network

2620/32asyncTo the NOC

Out of Band Ethernet

NetFlowCollector

NetFlowenabledrouters

Routerconsoles


Backbone Network Design


Backbone Design

• Routed Backbone• Switched Backbone• Leased point-to-point circuits

nx64K, T1/E1, T3/E3, OC3, OC12,...

• ATM/Frame Relay service from telcoT3, OC3, OC12,… deliveryeasily upgradeable bandwidth (CIR)

5


Distributed Network Design

• PoP design “standardised”operational scalability and simplicity

• ISP essential services distributed aroundbackbone

• NOC and “backup” NOC

• Redundant backbone links


Distributed Network Design

POP One

POP Two

POP Three

Customerconnections

Customerconnections

Customerconnections

Externalconnections

Externalconnections Operations Centre

BackupOperations Centre

ISP Services

ISP Services

ISP Services


Backbone Links

• ATM/Frame Relaynow less popular due to overhead, extra equipment,and shared with other customers of the telco

• Leased Linemore popular with backbone providers

IP over Optics and MPLS coming into the mainstream


Long Distance Backbone Links

• Tend to cost more• Plan for the future (at least two years ahead)

but stay in budgetUnplanned “emergency” upgrades can be disruptivewithout redundancy

• Allow sufficient capacity on alternative pathsfor failure situations

sufficient can be 20% to 50%


Long Distance Links

POP One

POP Two

POP Three

Long distance link

Alternative/Backup Path


Metropolitan Area Backbone Links

• Tend to be cheaperCircuit concentrationChoose from multiple suppliers

• Think bigMore redundancyLess impact of upgradesLess impact of failures

6


Metropolitan Area Backbone Links

POP One

POP Two

POP Three

Metropolitan Links

Metropolitan Links

Traditional Point to Point Links32© 2005, Cisco Systems, Inc. All rights reserved.

Cisco ISPWorkshops

ISP Services

DNS, Mail, Newsdesign and location


ISP Services:DNS

• Domain Name SystemProvides name and address resolutionServers need to be differentiated, properlylocated and specified

Primary nameserverSecondary nameserverCaching nameserver – resolver


ISP Services:DNS

• Primary nameserverHolds ISP zone files

forward zone (list of name to address mappings) for allISP’s and any customer zonesreverse zone (list of address to name mappings) for allISP’s address space

One Unix server, fast I/O, reasonable amount ofmemory (512Mbytes), reasonable diskLocated in secure part of net, e.g. NOC LAN


ISP Services:DNS

• Secondary nameserverHolds copies of ISP zone filesAt least two are required, more is betterUnix server, fast I/O, reasonable amount of memory(512Mbytes), reasonable diskShould be geographically separate from each otherand the primary DNS

At different PoPsOn a different continent e.g. www.secondary.comAt another ISP


ISP Services:Secondary DNS Example

• apnic.net zoneprimary DNS in Brisbanesecondary DNS around the world

$ dig apnic.net ns

;; ANSWER SECTION:apnic.net. 50m44s IN NS svc00.apnic.net.apnic.net. 50m44s IN NS ns.ripe.net.apnic.net. 50m44s IN NS rs.arin.net.apnic.net. 50m44s IN NS ns.apnic.net.

;; ADDITIONAL SECTION:svc00.apnic.net. 1d23h53m25s IN A 202.12.28.131ns.ripe.net. 1d23h54m46s IN A 193.0.0.193rs.arin.net. 1d23h53m25s IN A 192.149.252.21ns.apnic.net. 1d9h29m16s IN A 203.37.255.97

Tokyo

Amsterdam

Washington

Brisbane

7


ISP Services:Secondary DNS Example

• apnic.net zoneprimary DNS in Brisbane (ns.apnic.net)secondary DNS run by APNIC in Tokyo(svc00.apnic.net)zone secondaried by

RIPE NCC in AmsterdamARIN in Washington

Geographical and service provider redundancy – thisis the perfect example!


ISP Services:DNS

• Caching nameserverThis is the resolver – it is the DNS cacheYour customers use this as resolver, NOT your primaryor secondary DNSProvides very fast lookupsDoes NOT secondary any zonesOne, or preferably two per PoP (redundancy)Unix server, fast I/O, large amount of memory(512Mbytes+ depending on number of zones)


ISP Services:Caching Nameserver

To Core Routers

DIAL network

Web Cache

DNS Cache DNS Cache

Radius proxy

Switch redundancyRouter redundancyDNS Cache redundancy

DIAL users automatically given the IP addressesof DNS caches when they dial in


ISP Services:Anycasting the Caching Nameserver

• One trick of the tradeassign two unique IP addresses to befor the two DNS resolver systems

use these two IP addresses in every PoProute the two /32s across your backboneeven if the two resolver systems in the local PoP aredown, the IGP will ensure that the next nearestresolvers will be reachableKnown as IP Anycast

GeekAlert


ISP Services:DNS

• Efficient and resilient designPrimary DNS – keep it secureSecondary DNS – geographical and providerredundancy

Don’t ever put them on the same LAN, switched orotherwiseDon’t put them in the same PoP

Caching DNS – one or two per PoPreduces DNS traffic across backbonemore efficient, spreads the load


ISP Services:DNS

• SoftwareMake sure that the BIND distribution on the Unix systemis up to date

the vendor’s distribution is rarely current

Pay attention to bug reports, security issuesReboot the DNS cache on a regular (e.g. monthly) basis

clears out the cachereleases any lost RAMaccepted good practice by system administrators

8


ISP Services:DNS

• ImplementationPut all your hosts, point-to-point links and loopbacksinto the DNS

under your ISP’s domain nameuse sensible/meaningful names

Put all your hosts, point-to-point links and loopbacksinto the REVERSE DNS also

don’t forget about in-addr.arpa – many ISPs dosome systems demand forward/reverse DNS mappingbefore allowing access


ISP Services:Mail

• Must have at least two mail hosts (MX records) forall supported domains

geographical separation helps

• POP3 server dedicated to that functionDIAL users get mail from here

• SMTP gateway dedicated to that functionDIAL users send mail via here

• Mail relay open to CUSTOMERS only!


ISP Services:Mail Example

• telstra.net mail (MX records)primary MX is mako1backup MX is postoffice – two addressesbackup MX used if primary unavailable

$ dig telstra.net mx

;; ANSWER SECTION:telstra.net. 1H IN MX 10 postoffice.telstra.net.telstra.net. 1H IN MX 5 mako1.telstra.net.

;; ADDITIONAL SECTION:postoffice.telstra.net. 1H IN A 139.130.4.7postoffice.telstra.net. 1H IN A 203.50.1.76mako1.telstra.net. 1H IN A 203.50.0.28


ISP Services:Mail

• SoftwareMake sure that the MAIL and POP3 distributionson the Unix system are up to date

the vendor’s distribution are rarely current

Pay attention to bug reports, security issues,unsolicited junk mail complaints

IMPORTANT: Do NOT allow non-customers to use your mail system as a relay


ISP Services:News

• News servers provide a Usenet news feed tocustomers

• Distributed design requiredIncoming newsfeed to one large serverDistributed to feed servers in each PoPFeed servers provide news feed to customersOutgoing news goes to another serverSeparate reading news systemSeparate posting news system


ISP Services:News System Placement

POP One

POP Two

POP Three

Customerconnections

Customerconnections

Customerconnections

Externalconnections

Externalconnections News Collector

News Feeder

News Feeder

News Feeder

News Distributor

9


ISP Services:News System Placement

POP One

POP Two

POP Three

Customerconnections

Customerconnections

Customerconnections

Externalconnections

Externalconnections News Collector

News Feeder

News Feeder

News Feeder

News Distributor505050© 2005, Cisco Systems, Inc. All rights reserved.

Cisco ISPWorkshops

ISP Services:News

• SoftwareMake sure that the Internet News distribution onthe Unix system is up to date

the vendor’s distribution is rarely current

Pay attention to bug reports, security issues,unsolicited junk posting complaints

IMPORTANT: Do NOT allow non-customers to use your news system for posting messages


Addressing


Where to get IP addresses and AS numbers

• Your upstream ISP• Africa

AfriNIC – http://www.afrinic.net

• Asia and the PacificAPNIC – http://www.apnic.net

• North AmericaARIN – http://www.arin.net

• Latin America and the CaribbeanLACNIC – http://www.lacnic.net

• Europe and Middle EastRIPE NCC – http://www.ripe.net


ARIN

Internet Registry Regions

LACNIC


Getting IP address space

• Take part of upstream ISP’s PA spaceor

• Become a member of your Regional InternetRegistry and get your own allocation

Require a plan for a year aheadGeneral policies are outlined in RFC2050, more specificdetails are on the individual RIR website

• There is plenty of IPv4 address spaceregistries require high quality documentation

10


Addressing Plans – ISP Infrastructure

• Address block for router loop-backinterfaces

• Address block for infrastructureper PoP or whole backbonesummarise between sites if it makes senseallocate according to genuine requirements,not historic classful boundaries


Addressing Plans – Customer

• Customers assigned address spaceaccording to need

• Should not be reserved or assigned on aper PoP basis

ISP iBGP carries customer netsaggregation not required and usually notdesirable


Addressing Plans – ISP Infrastructure

Phase One220.10.0.0/21

Customer assignments Instrastructure Loopbacks

/24220.10.6.255220.10.0.1

220.10.0.0/20

Original assignments New Assignments

/24/24220.10.0.1

220.10.5.255 220.10.15.255

Phase Two


Addressing PlansPlanning

• Registries will usually allocate the nextblock to be contiguous with the firstallocation

Minimum allocation is /21

Very likely that subsequent allocation willmake this up to a /20So plan accordingly


Addressing Plans (contd)

• Document infrastructure allocationeases operation, debugging and management

• Document customer allocationcontained in iBGPeases operation, debugging and managementsubmit network object to RIR Database


Routing Protocols

11


Routing Protocols

• IGP – Interior Gateway Protocolcarries infrastructure addresses, point-to-point linksexamples are OSPF, ISIS, EIGRP...

• EGP – Exterior Gateway Protocolcarries customer prefixes and Internet routescurrent EGP is BGP version 4

• No link between IGP and EGP


Why Do We Need an IGP?

• ISP backbone scalingHierarchyModular infrastructure constructionLimiting scope of failure

Healing of infrastructure faults using dynamicrouting with fast convergence


Why Do We Need an EGP?

• Scaling to large networkHierarchyLimit scope of failure

• PolicyControl reachability to prefixesMerge separate organizationsConnect multiple IGPs


Interior versus Exterior Routing Protocols

• Interiorautomatic neighbourdiscoverygenerally trust your IGProuters

prefixes go to all IGProutersbinds routers in one AStogether

• Exteriorspecifically configuredpeersconnecting withoutside networks

set administrativeboundariesbinds AS’s together


Interior versus Exterior Routing Protocols

• InteriorCarries ISPinfrastructureaddresses onlyISPs aim to keep theIGP small forefficiency andscalability

• ExteriorCarries customerprefixesCarries Internet prefixesEGPs are independentof ISP network topology


Hierarchy of Routing Protocols

BGP4and OSPF/ISIS

FDDI

Other ISPs

CustomersLocalIXP

BGP4 Static/BGP4

BGP4

12


Routing Protocols:Choosing an IGP

• Review the “Introduction to Link StateProtocols” presentation

i.e. – OSPF and ISIS have very similar properties

• ISP usually chooses between OSPF and ISISChoose which is appropriate for your operators’experienceIn IOS, both OSPF and ISIS have sufficient “nerdknobs” to tweak the IGP’s behaviour


Routing Protocols:IGP Recommendations

• Keep the IGP routing table as small as possibleIf you can count the routers and the point to point linksin the backbone, that total is the number of IGP entriesyou should see

• IGP details:Should only have router loopbacks, backbone WANpoint-to-point link addresses, and network addressesof any LANs having an IGP running on themStrongly recommended to use inter-routerauthenticationUse inter-area summarisation if possible


Routing Protocols:More IGP recommendations

• To fine tune IGP table size more, consider:Using “ip unnumbered” on customer point-to-pointlinks – saves carrying that /30 in IGP

(If customer point-to-point /30 is required formonitoring purposes, then put this in iBGP)Use contiguous addresses for backbone WAN links ineach area – can then summarise into backbone areaDon’t summarise router loopback addresses – as iBGPneeds thoseUse iBGP for carrying anything which does notcontribute to the Link State Routing process


Routing Protocols:iBGP Recommendations

• iBGP should carry everything whichdoesn’t contribute to the IGP routingprocess

Internet routing tableCustomer assigned addresses

Customer point-to-point linksDIAL network pools, passive LANs, etc


Routing Protocols:More iBGP Recommendations

• Scalable iBGP features:Use neighbour authentication

Use peer-groups to speed update process andfor configuration efficiencyUse communities for ease of filtering

Use route-reflector hierarchyRoute reflector pair per PoP (overlaid clusters)

Use route flap damping at the network edges


Security

13


Security

• ISP Infrastructure security• ISP Network security• Security is not optional!• ISPs need to:

protect themselveshelp protect their customers from the Internetprotect the Internet from their customers

• The following slides are generalrecommendations

do more research on security before deploying anynetwork


ISP Infrastructure Security

• router securityusernames, passwords, vty filters, TACACS+Disable telnet on vtys, only use SSHvty filters should only allow NOC access, noexternal accessSee IOS Essentials for the recommendedpractices for ISPs



• ISP server securityusernames, passwords, TCP wrappers, IPTABLESprotect all servers using routers with strong filtersapplied

• Hosted services securityprotect network from hosted servers using routerswith strong filtersprotect hosted servers from Internet using routers withstrong filters


ISP Infrastructure SecurityISP Server Protection

DNScache

DNSsecondary POP3 Mail

Relay NEWS

To core routers


Access-list examples:

Allow tcp/established to all serversICMPDNS 2ary: udp/53 and tcp/53POP3: tcp/110Mail Relay: tcp/25 and ISP address

range onlyNews: tcp/119 and ISP

address range onlyDNS Cache: udp/53Web server: tcp/80

Other necessary filters:

All servers: SSH (tcp/22) from NOC LAN only

Webserver


Access-list examples:InboundAllow tcp/established to all serversICMPWeb server: tcp/80SSH for customer accessAny other ports for services

sold to customers

OutboundICMPAllow DNS udp/53 and

tcp/53Block all access to ISP

address range

ISP Infrastructure SecurityHosted Server Protection

Server5Server1 Server2 Server3 Server4

To core routers


Server6



• premises securitylocks – electronic/card key preferredsecure access – 24x7 security arrangementsenvironment control – good aircon

• staff responsibilitypassword policy, strangers, temp staffemployee exit procedures

• RFC2196(Site Security Handbook)

• RFC3871(Operational Security Requirements for Large ISP IPNetwork Infrastructure )

14


ISP Network Security

• Denial of Service Attackseg: “smurfing”see http://www.denialinfo.com

• Effective filteringnetwork borders – see Cisco ISP Essentialscustomer connections – unicast RPFnetwork operation centreISP corporate network – behind firewall


ISP Network SecuritySecure external access

• How to provide staff access from outsideset up ssh gateway (Unix system with ssh daemon andnothing else configured)provide ssh client on all staff laptopsssh available on Unix and Windowsssh is Secure Shell – encrypted link

• How not to provide access from outsidetelnet, rsh, rlogin – these are all insecureopen host – insecure, can be compromised


Ingress & Egress Route Filtering

Your customers should not besending any IP packets out to the

Internet with a source addressother then the address you have

allocated to them!


Out of Band Management



• Not optional!• Allows access to network equipment in times of

failure

• Ensures quality of service to customersminimises downtimeminimises repair timeeases diagnostics and debugging



• OoB Example – Access server:modem attached to allow NOC dial inconsole ports of all network equipment connected toserial portsLAN and/or WAN link connects to network core, or viaseparate management link to NOC

• Full remote control access under allcircumstances

15


Out of Band Network

Ethernetto the NOC

Router, switchand ISP server

consoles

(Optional) Out of bandWAN link to other PoPs

Modem – accessto PSTN for out of

band dialin

Equipment RackEquipment Rack



• OoB Example – Statistics gathering:Routers are NetFlow and syslog enabled

Management data is congestion/failure sensitiveEnsures management data integrity in case of failure

• Full remote information under all circumstances


Test Laboratory


Test Laboratory

• Designed to look like a typical PoPoperated like a typical PoP

• Used to trial new services or newsoftware under realistic conditions

• Allows discovery and fixing of potentialproblems before they are introduced tothe network


Test Laboratory

• Some ISPs dedicate equipment to the lab

• Other ISPs “purchase ahead” so thattoday’s lab equipment becomestomorrow’s PoP equipment

• Other ISPs use lab equipment for “hotspares” in the event of hardware failure


Test Laboratory

• Can’t afford a test lab?Set aside one spare router and server to trial new services

Never ever try out new hardware, software or services onthe live network

• Every major ISP in the US and Europe has a test labIt’s a serious consideration

16


Operational Considerations


Operational Considerations

Why design the world’s best networkwhen you have not thought about whatoperational good practices should be

implemented?


Operational ConsiderationsMaintenance

• Never work on the live network, no matter howtrivial the modification may seem

Establish maintenance periods which your customers areaware of

e.g. Tuesday 4-7am, Thursday 4-7am

• Never do maintenance on a FridayUnless you want to work all weekend cleaning up

• Never do maintenance on a MondayUnless you want to work all weekend preparing


Operational ConsiderationsSupport

• Differentiate between customer support and theNetwork Operations Centre

Customer support fixes customer problemsNOC deals with and fixes backbone and Internet relatedproblems

• Network Engineering team is last resortthey design the next generation network, improve therouting design, implement new services, etcthey do not and should not be doing support!


Operational ConsiderationsNOC Communications

• NOC should know contact details forequivalent NOCs in upstream providersand peers

• Or consider joining the INOC-DBA systemVoice over IP phone system using SIPRuns over the Internetwww.pch.net/inoc-dba for more information


ISP Network Design

Summary

17


ISP Design Summary

• KEEP IT SIMPLE & STUPID ! (KISS)

• Simple is elegant is scalable

• Use Redundancy, Security, andTechnology to make life easier for yourself

• Above all, ensure quality of service foryour customers


ISP Network DesignISP/IXP WorkshopsISP/IXP Workshops

Date post:	02-May-2020
Category:	Documents
Upload:	others
View:	16 times
Download:	0 times

ISP Network Design - Network Startup Resource Center … · ISP Network Design ISP/IXP Workshops...

Documents