+ All Categories
Transcript
Page 1: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

Networking

Nick FeamsterGeorgia Tech

2

Goal of This Tutorial

bull Teach engineers the basics of networking and ISP operations

bull Networks todayndash Business modelsndash Operations (NOC operators)

bull Common problemsbull Measurement Monitoring and Security

3

Todayrsquos Networks

bull Service provider business models

bull Network operations center

bull Network operators and engineers

4

Business Models

bull Increasingly commoditized (see Geoff Hustonrsquos talk at NANOG)

bull Status quo Establish transit costs bill at 95th percentile of usage

bull Future differential pricing preference for certain groups of users applications

5

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

6

Net Neutrality

7

Network Operations

bull Operators run the day-to-day operations of the networkndash Adjusting to shifts in traffic failures etcndash Responding to security threatsndash Provisioning new customers

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 2: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

2

Goal of This Tutorial

bull Teach engineers the basics of networking and ISP operations

bull Networks todayndash Business modelsndash Operations (NOC operators)

bull Common problemsbull Measurement Monitoring and Security

3

Todayrsquos Networks

bull Service provider business models

bull Network operations center

bull Network operators and engineers

4

Business Models

bull Increasingly commoditized (see Geoff Hustonrsquos talk at NANOG)

bull Status quo Establish transit costs bill at 95th percentile of usage

bull Future differential pricing preference for certain groups of users applications

5

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

6

Net Neutrality

7

Network Operations

bull Operators run the day-to-day operations of the networkndash Adjusting to shifts in traffic failures etcndash Responding to security threatsndash Provisioning new customers

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 3: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

3

Todayrsquos Networks

bull Service provider business models

bull Network operations center

bull Network operators and engineers

4

Business Models

bull Increasingly commoditized (see Geoff Hustonrsquos talk at NANOG)

bull Status quo Establish transit costs bill at 95th percentile of usage

bull Future differential pricing preference for certain groups of users applications

5

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

6

Net Neutrality

7

Network Operations

bull Operators run the day-to-day operations of the networkndash Adjusting to shifts in traffic failures etcndash Responding to security threatsndash Provisioning new customers

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 4: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

4

Business Models

bull Increasingly commoditized (see Geoff Hustonrsquos talk at NANOG)

bull Status quo Establish transit costs bill at 95th percentile of usage

bull Future differential pricing preference for certain groups of users applications

5

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

6

Net Neutrality

7

Network Operations

bull Operators run the day-to-day operations of the networkndash Adjusting to shifts in traffic failures etcndash Responding to security threatsndash Provisioning new customers

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 5: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

5

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

6

Net Neutrality

7

Network Operations

bull Operators run the day-to-day operations of the networkndash Adjusting to shifts in traffic failures etcndash Responding to security threatsndash Provisioning new customers

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 6: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

6

Net Neutrality

7

Network Operations

bull Operators run the day-to-day operations of the networkndash Adjusting to shifts in traffic failures etcndash Responding to security threatsndash Provisioning new customers

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 7: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

7

Network Operations

bull Operators run the day-to-day operations of the networkndash Adjusting to shifts in traffic failures etcndash Responding to security threatsndash Provisioning new customers

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 8: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

8

Point-of-Presence (PoP)

bull A ldquoclusterrdquo of routers in a single physical location

bull Inter-PoP linksndash Long distances

ndash High bandwidth

bull Intra-PoP linksndash Cables between racks or floors

ndash Aggregated bandwidth

PoP

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 9: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

9

Example Abilene Network Topology

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 10: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

10

Another Example Backbone

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 11: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

11

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 12: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

12

Internet Routing Protocol BGP

Route Advertisement

Autonomous Systems (ASes)

Session

Traffic

Destination Next-hop AS Path1302070016

1302070016

19258989

6625025244

105782637

174hellip 2637

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 13: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

13

Two Flavors of BGP

bull External BGP (eBGP) exchanging routes between ASes

bull Internal BGP (iBGP) disseminating routes to external destinations among the routers within an AS

eBGPiBGP

Question Whatrsquos the difference between IGP and iBGP

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 14: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

14

IPv4 Addresses Networks of Networks

bull 32-bit number in ldquodotted-quadrdquo notation

ndash wwwccgatechedu --- 130207736

10000010 11001111 00000111 00100100

Network (16 bits) Host (16 bits)

130 207 7 36

bull Problem 232 addresses is a lot of table entries

bull Solution Routing based on network and host

ndash 1302070016 is a 16-bit prefix with 216 IP addresses

Topological Addressing

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 15: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

15

Pre-1994 Classful Addressing

Network ID Host ID

8 16

Class A

32

0

Class B 10

Class C 110

Multicast AddressesClass D 1110

Reserved for experimentsClass E 1111

24

8 blocks (eg MIT has 180008)

16 blocks (eg Georgia Tech has 1302070016)

24 blocks (eg ATampT Labs has 19220225024)

Simple Forwarding Address range specifies network ID length

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 16: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

16

Classless Interdomain Routing (CIDR)

IP Address 65142480 ldquoMaskrdquo 2552552520

01000001 00001110 11111000 00000000

11111111 11111111 11111100 00000000

Use two 32-bit numbers to represent a network Network number = IP address + Mask

Example BellSouth Prefix 6514248022

Address no longer specifies network ID rangeNew forwarding trick Longest Prefix Match

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 17: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

17

Benefits of CIDR

bull Efficiency Can allocate blocks of prefixes on a finer granularity

bull Hierarchy Prefixes can be aggregated into supernets (Not always done Typically not in fact)

Customer 1

Customer 2

ATampT Internet

1220249024

1220231024120008

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 18: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

18

Growth of IP Prefixes

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 19: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

19

1994-1998 Linear Growth

bull About 10000 new entries per yearbull In theory less instability at the edges (why)

Source Geoff Huston

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 20: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

20

Around 2000 Fast Growth Resumes

Claim remaining 8s will be exhausted within the next 5-10 years

T Hain ldquoA Pragmatic Report on IPv4 Address Space Consumptionrdquo Cisco IPJ September 2005

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 21: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

21

Fast growth resumes

Rapid growth in routing tables

Dot-Bomb Hiccup

Significant contributor Multihoming

Source Geoff Huston

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 22: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

22

The Address Allocation Process

bull Allocation policies of RIRs affect pressure on IPv4 address space

IANA

AfriNIC APNIC ARIN LACNIC RIPE

httpwwwianaorgassignmentsipv4-address-space

Georgia Tech

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 23: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

23

Common Problems

bull Diagnosis and troubleshooting (hence measurement)bull Traffic engineeringbull Securitybull Design and capacity planning

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 24: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

24

What can go wrong

Two-thirds of the problems are caused by configuration of the routing protocol

Some downtime is very hard to protect againsthellip

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 25: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

25

Measurement and Monitoring

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 26: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

26

Passive vs Active Measurement

bull Passive Measurement Collection of packets flow statistics of traffic that is already flowing on the networkndash Packet tracesndash Flow statisticsndash Application-level logs

bull Active Measurement Inject ldquoprobingrdquo traffic to measure various characteristicsndash Traceroutendash Pingndash Application-level probes (eg Web downloads)

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 27: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

27

Billing for Internet Usage

bull 95th Percentile billingndash Customer network pays for ldquocommitted information

raterdquo (CIR)ndash Throughput measured every 5 minutes (typically with

SNMP flow statistics also can be used for billing)ndash Customer billed based on 95th percentile

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 28: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

28

Passive Traffic Data Measurement

bull SNMP bytepacket counts everywhere

bull Packet monitoring selected locations

bull Flow monitoring typically at edges (if possible)ndash Direct computation of the traffic matrixndash Input to denial-of-service attack detection

bull Deep Packet Inspection also at edge where possible

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 29: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

29

Simple Network Management Protocol

bull Management Information Base (MIB)ndash Information storendash Unique variables named by

OIDsndash Accessed with SNMP

bull Specific MIBs for bytepacket counts (per link)

Manager Agent

SNMP

DB

ManagedObjects

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 30: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

30

SNMP (Passive)

bull Advantage ubiquitousndash Supported on all networking equipmentndash Multiple products for polling and analyzing data

bull Disadvantages ndash Coarse granularityndash Cannot express complex queries on the datandash Unreliable delivery of the data using UDP

bull Utilityndash Link utilization (billing)ndash Traffic matrix inference

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 31: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

31

Packet-level Monitoring

bull Passive monitoring to collect full packet contents (or at least headers)

bull Advantages lots of detailed informationndash Precise timing informationndash Information in packet headers

bull Disadvantages overheadndash Hard to keep up with high-speed linksndash Often requires a separate monitoring device

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 32: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

32

Full Packet Capture (Passive)

Example Georgia Tech OC3Mon

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 33: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

33

What is a flow

bull Source IP addressbull Destination IP addressbull Source portbull Destination portbull Layer 3 protocol typebull TOS byte (DSCP)bull Input logical interface (ifIndex)

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 34: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

34

Cisco NetFlow

bull Basic output ldquoFlow recordrdquondash Most common version is v5

bull Current version (9) is being standardized in the IETF (template-based)ndash More flexible record formatndash Much easier to add new flow record types

Core Network

Collection and Aggregation

Collector (PC)Approximately 1500 bytes

20-50 flow recordsSent more frequently if traffic increases

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 35: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

35

Flow Record Contents

bull Source and Destination IP address and portbull Packet and byte countsbull Start and end timesbull ToS TCP flags

Basic information about the flowhellip

hellipplus information related to routing

bull Next-hop IP addressbull Source and destination ASbull Source and destination prefix

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 36: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

36

flow 1 flow 2 flow 3 flow 4

Aggregating Packets into Flows

bull Criteria 1 Set of packets that ldquobelong togetherrdquondash Sourcedestination IP addresses and port numbersndash Same protocol ToS bits hellip ndash Same inputoutput interfaces at a router (if known)

bull Criteria 2 Packets that are ldquocloserdquo together in timendash Maximum inter-packet spacing (eg 15 sec 30 sec)ndash Example flows 2 and 4 are different flows due to time

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 37: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

37

Reducing Measurement Overhead

bull Filtering on interfacendash destination prefix for a customerndash port number for an application (eg 80 for Web)

bull Sampling before insertion into flow cachendash Random deterministic or hash-based samplingndash 1-out-of-n or stratified based on packetflow sizendash Two types packet-level and flow-level

bull Aggregation after cache evictionndash packetsflows with same next-hop ASndash packetsflows destined to a particular service

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 38: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

38

Packet Sampling for Flow Monitoring

bull Packet sampling before flow creation (Sampled Netflow)ndash 1-out-of-m sampling of individual packets (eg m=100)ndash Create of flow records over the sampled packets

bull Reducing overheadndash Avoid per-packet overhead on (m-1)m packetsndash Avoid creating records for a large number of small flows

bull Increasing overhead (in some cases)ndash May split some long transfers into multiple flow records ndash hellip due to larger time gaps between successive packets

time

not sampled

two flowstimeout

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 39: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

39

Sampling Flow-Level Sampling

bull Sampling of flow records evicted from flow cachendash When evicting flows from table or when analyzing flows

bull Stratified sampling to put weight on ldquoheavyrdquo flowsndash Select all long flows and sample the short flows

bull Reduces the number of flow records ndash Still measures the vast majority of the traffic

Flow 1 40 bytesFlow 2 15580 bytesFlow 3 8196 bytesFlow 4 5350789 bytesFlow 5 532 bytesFlow 6 7432 bytes

sample with 100 probability

sample with 01 probability

sample with 10 probability

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 40: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

40

Two Main Approaches

bull Packet-level Monitoringndash Keep packet-level statisticsndash Examine (and potentially log) variety of packet-level

statistics Essentially anything in the packetndash Timing

bull Flow-level Monitoringndash Monitor packet-by-packet (though sometimes

sampled)ndash Keep aggregate statistics on a flow

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 41: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

41

Packet Capture on High-Speed Links

Example Georgia Tech ldquoOC3Monrdquo

bull Rack-mounted PCbull Optical splitterbull Data Acquisition and

Generation (DAG) card

Source endacecom

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 42: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

42

Characteristics of Packet Capture

bull Allows inspection on every packet on 10G links

bull Disadvantagesndash Costlyndash Requires splitting optical fibersndash Must be able to filterstore data

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 43: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

43

Data Measurement Repositories

bull AbileneInternet 2 Observatoryndash Configuration examplesndash SNMP datandash ISIS BGP routing data NetFlow traffic data

bull RouteViewsndash BGP updatesndash BGP table snapshots

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 44: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

44

Multihoming and Traffic Engineering

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 45: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

45

What is Multihoming

bull The use of redundant network links for the purposes of external connectivity

bull Can be achieved at many layers of the protocol stack and many places in the networkndash Multiple network interfaces in a PC

ndash An ISP with multiple upstream interfaces

bull Can refer to having multiple connections tondash The same ISP

ndash Multiple ISPs

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 46: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

46

Why Multihome

bull Redundancybull Availabilitybull Performancebull Cost

Interdomain traffic engineering the process by which a multihomed network configures its

network to achieve these goals

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 47: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

47

Redundancy

bull Maintain connectivity in the face ofndash Physical connectivity problems (fiber cut device

failures etc)ndash Failures in upstream ISP

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 48: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

48

Performance

bull Use multiple network links at once to achieve higher throughput than just over a single link

bull Allows incoming traffic to be load-balanced

70 of traffic30 of traffic

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 49: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

49

Multihoming in IP Networks Today

bull Stub AS no transit service for other ASesndash No need to use BGP

bull Multi-homed stub AS has connectivity to multiple immediate upstream ISPsndash Need BGPndash No need for a public AS numberndash No need for IP prefix allocation

bull Multi-homed transit AS connectivity to multiple ASes and transit servicendash Need BGP public AS number IP prefix allocation

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 50: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

50

BGP or no

bull Advantages of static routingndash Cheapersmaller routers (less true nowadays)ndash Simpler to configure

bull Advantages of BGPndash More control of your destiny (have providers stop

announcing you)ndash Fastermore intelligent selection of where to send

outbound packetsndash Better debugging of net problems (you can see the

Internet topology now)

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 51: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

51

Same Provider or Multiple

bull If your provider is reliable and fast and affordably and offers good tech-support you may want to multi-home initially to them via some backup path (slow is better than dead)

bull Eventually yoursquoll want to multi-home to different providers to avoid failure modes due to one providerrsquos architecture decisions

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 52: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

52

Multihomed Stub One Link

bull Downstream ISPrsquos routers configure default (ldquostaticrdquo) routes pointing to border router

bull Upstream ISP advertises reachability

Upstream ISP

Multiple links between same pair of routers

Default routes to ldquoborderrdquo

ldquoStubrdquoISP

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 53: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

53

Multihomed Stub Multiple Links

bull Use BGP to share loadbull Use private AS number (why is this OK)bull As before upstream ISP advertises prefix

Upstream ISP

Multiple links to different upstream routers

ldquoStubrdquoISP

Internal routing for ldquohot potatordquo

BGP for load balance at edge

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 54: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

54

Multihomed Stub Multiple ISPs

bull Many possibilitiesndash Load sharingndash Primary-backupndash Selective use of different ISPs

bull Requires BGP public AS number etc

ldquoStubrdquoISP

Upstream

ISP 1

Upstream

ISP 2

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 55: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

55

Multihomed Transit Network

bull BGP everywherebull Incoming and outcoming trafficbull Challenge balancing load on intradomain and egress

links given an offered traffic load

TransitISP

ISP 1

ISP 2

ISP 3

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 56: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

56

Interdomain Traffic Engineering

bull The process by which a network operator configures the network to achievendash Traffic load balancendash Redundancy (primarybackup) etc

bull Two tasksndash Outbound traffic controlndash Inbound traffic control

bull Key Problems Predictability and Scalability

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 57: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

57

Outbound Traffic Control

bull Easier to control than inbound trafficndash Destination-based routing sender determines where

the packets go

bull Control over next-hop AS onlyndash Cannot control selection of the entire path

Provider 1 Provider 2

Control with local preference

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 58: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

58

Outbound Traffic Load Balancingbull Control routes to provider per-prefix

ndash Assign local preference across destination prefixesndash Change the local preference assignments over time

bull Useful inputs to load balancingndash End-to-end path performance datandash Outbound traffic statistics per destination prefix

bull Challenge Getting from traffic volumes to groups of prefixes that should be assigned to each link

Premise of ldquointelligent route controlrdquo preoducts

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 59: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

59

Traffic Engineering Goals

bull Predictabilityndash Ensure the BGP decision process is deterministicndash Assume that BGP updates are (relatively) stable

bull Limit overhead introduced by routing changesndash Minimize frequency of changes to routing policiesndash Limit number of prefixes affected by changes

bull Limit impact on how traffic enters the networkndash Avoid new routes that might change neighborrsquos mindndash Select route with same attributes or at least path length

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 60: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

60

Managing Scalebull Destination prefixes

ndash More than 90000 destination prefixes

bull Donrsquot want to have per-prefix routing policies

ndash Small fraction of prefixes contribute most of the traffic

bull Focus on the small number of heavy hitters

ndash Define routing policies for selected prefixes

bull Routing choicesndash About 27000 unique ldquorouting choicesrdquo

bull Help in reducing the scale of the problem

ndash Small fraction of ldquorouting choicesrdquo contribute most traffic

bull Focus on the very small number of ldquorouting choicesrdquo

ndash Define routing policies on common attributes

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 61: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

61

Achieving Predictability

bull Route prediction with static analysisndash Helpful to know effects before deploymentndash Static analysis can help

TopologyBGP policy

configuration

eBGP routes

Offered traffic

BGP routingmodel

Flow of traffic through the network

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 62: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

62

Challenges to Predictabilitybull For transit ISPs effects on incoming traffic

ndash Lack of coordination strikes again

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 63: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

63

ldquoHot Potatordquo routing

Inter-AS Negotiation

bull Coordination aids predictabilityndash Negotiate where to sendndash Inbound and outboundndash Mutual benefits

bull How to implementndash What info to exchangendash Protecting privacyndash How to prioritize choicesndash How to prevent cheating

Destination 2

Destination 1

multiplepeeringpoints

Provider A

Provider B

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 64: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

64

Outbound Multihoming Goals

bull Redundancyndash Dynamic routing will failover to backup link

bull Performancendash Select provider with best performance per prefix

ndash Requires active probing

bull Costndash Select provider per prefix over time to minimize the

total financial cost

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 65: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

65

Inbound Traffic Control

bull More difficult no control over neighborsrsquo decisions

bull Three common techniques (previously discussed)ndash AS path prependingndash Communities and local preferencendash Prefix splitting

How does todayrsquos paper (MONET) control inbound traffic

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 66: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

66

How many links are enough

K upstream ISPs

Not much benefit beyond 4 ISPs

Akella et al ldquoPerformance Benefits of Multihomingrdquo SIGCOMM 2003

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 67: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

67

Problems with Multihoming in IPv4

bull Routing table growthndash Provider-based addressingndash Advertising prefix out multiple ISPs ndash canrsquot aggregate

bull Poor control over inbound trafficndash Existing mechanisms do not allow hosts to control

inbound traffic

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 68: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

68

GeorgiaTech

Internet Routing Overview

bull Intradomain (ie ldquointra-ASrdquo) routingbull Interdomain routing

Comcast

Abilene

ATampT Cogent

Autonomous Systems (ASes)

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 69: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

69

Configuration Problems ldquoAS 7007rdquoldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

UUNet

Florida InternetBarn

Sprint

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 70: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

70

Diagnosis and Troubleshootingldquohellipa glitch at a small ISPhellip triggered a major outage in Internet access across the country The problem started when MAI Network Servicespassed bad router information from one of its customers onto Sprintrdquo

-- newscom April 25 1997

ldquoMicrosofts websites were offline for up to 23 hoursbecause of a [router] misconfigurationhellipit took nearly a day to determine what was wrong and undo the changesrdquo -- wiredcom January 25 2001

ldquoWorldCom Inchellipsuffered a widespread outage on its Internet backbone that affected roughly 20 percent of its US customer base The network problemshellipaffected millions of computer users worldwide A spokeswoman attributed the outage to a route table issue -- cnncom October 3 2002

A number of Covad customers went out from 5pm today due to supposedly a DDOS (distributed denial of service attack) on a key Level3 data center which later was described as a route leak (misconfiguration)ldquo

-- dslreportscom February 23 2004

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 71: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

71

Operator Mailing List (NANOG)Date Mon 18 Oct 2004 091515 -0700Subject Level 3 US east coast issues

Level 3 experiencing widespread unspecified routing issues on the US east coast Master ticket 1086844 Anyone have more specific information

Date Mon 18 Oct 2004 122034 -0400 (EDT)Subject Re Level 3 US east coast issues

Level 3 is currently experiencing a backbone outage causing routing instability and packet loss We are working to restore and will be sending hourly updateshellip

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 72: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

72

Operator Mailing List

0102030405060708090

Filtering RouteLeaks

RouteHijacks

RouteInstability

RoutingLoops

Blackholes

Occurr

ences o

ver

10 Y

ears

1995-1997 1998-2001 2001-2004

Note Only includes problems openly discussed on this list

Compare 83 power outages 1 fire

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 73: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

73

Routing Configuration

Ranking route selection

Dissemination internal route advertisement

Filtering route advertisement

Customer

Competitor

Primary

Backup

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 74: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

74

Internet Business Model (Simplified)

bull CustomerProvider One AS pays another for reachability to some set of destinations

bull ldquoSettlement-freerdquo Peering Bartering Two ASes exchange routes with one another

Provider

Peer

Customer

Preferences implemented with local preference manipulation

Destination

Pay to use

Get paid to use

Free to use

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 75: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

75

Peering Contracts Consistent Export

bull Rules of settlement-free peeringndash Advertise routes at all peering pointsndash Advertised routes must have equal ldquoAS path lengthrdquo

Sprint

ATampT

Enables ldquohot potatordquo routing

ldquoequally goodrdquoroutes

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 76: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

76

Consistent Export

bull Malicedeceptionbull iBGP signaling partitionbull Inconsistent export policy

neighbor 10123route-map PEER permit 10 set prepend 123

neighbor 10456route-map PEER permit 10 set prepend 123 123

Possible Causes

10123 456 1

10456 456 2

Neighbor AS Export

1 1 123

Export Clause Prepend

2 1 123 123

Two different Export Policies

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 77: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

77

Inconsistent Export in Practice

Feamster et al ldquoBorderGuard Detecting Cold Potatoes from Peersrdquo ACM IMC October 2004

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 78: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

78

Blackholes

Date Thu 18 Jul 2002 060510 -0400 (EDT)From Chad Oleary ltcolpoboxcomgtSubject Re problems with 701To ltnanogmeritedugt

Were starting to see the same issues with UUNet again Anyone elseseeing this Trying to reach Qwest

traceroute to 631461901 (631461901) 30 hops max 38 byte packets 1 esc-lp2-gwe-solutionscorpcom (631182201) 1167 ms 1163 ms 1142 ms 2 500Serial2-10GW1TPA2ALTERNET (1571301499) 1097 ms 1059 ms 1044 ms 3 161at-1-0-0XL4ATL1ALTERNET (1526381190) 13839 ms 14108 ms 16638 ms 4 0so-3-1-0XL2ATL5ALTERNET (152630238) 14370 ms 14587 ms 14553 ms 5 POS7-0BR2ATL5ALTERNET (1526382193) 13928 ms 14099 ms 14053 ms 6 7 hellip

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 79: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

79

Security

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 80: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

80

Security ldquoBogonrdquo Routes

Feamster et al ldquoAn Empirical Study of lsquoBogonrsquo Route Advertisementsrdquo ACM CCR January 2005

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 81: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

81

Spam Phishing etc

bull Unsolicited commercial emailbull As of about August 2008 estimates indicate that

about 95 of all email is spambull Common spam filtering techniques

ndash Content-based filtersndash DNS Blacklist (DNSBL) lookups Significant fraction of

todayrsquos DNS traffic

Can IP addresses from which spam is received be spoofed

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 82: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

82

Spam and Routing

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 83: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

83

Worms and Botnets

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 84: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

84

What is a Worm

bull Code that replicates and propagates across the networkndash Often carries a ldquopayloadrdquo

bull Usually spread via exploiting flaws in open servicesndash ldquoVirusesrdquo require user action to spread

bull First worm Robert Morris November 1988ndash 6-10 of all Internet hosts infected ()

bull Many more since but none on that scale until July 2001

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 85: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

85

Example Worm Code Red

bull Initial version July 13 2001

bull Exploited known ISAPI vulnerability in Microsoft IIS Web servers

bull 1st through 20th of each month spread20th through end of each month attack

bull Payload Web site defacementbull Scanning Random IP addressesbull Bug failure to seed random number generator

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 86: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

86

Code Red Revisions

bull Released July 19 2001

bull Payload flooding attack on wwwwhitehousegovndash Attack was mounted at the IP address of the Web site

bull Bug died after 20th of each month

bull Random number generator for IP scanning fixed

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 87: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

87

Code Red Host Infection Rate

Exponential infection rate

Measured using backscatter technique

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 88: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

88

Designing Fast-Spreading Worms

bull Hit-list scanningndash Time to infect first 10k hosts dominates infection timendash Solution Reconnaissance (stealthy scans etc)

bull Permutation scanningndash Observation Most scanning is redundantndash Idea Shared permutation of address space Start scanning

from own IP address Re-randomize when another infected machine is found

bull Internet-scale hit listsndash Flash worm complete infection within 30 seconds

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 89: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

89

Botnets

bull Bots Autonomous programs performing tasksbull Plenty of ldquobenignrdquo bots

ndash eg weatherbug

bull Botnets group of bots ndash Typically carries malicious connotationndash Large numbers of infected machinesndash Machines ldquoenlistedrdquo with infection vectors like worms

(last lecture)

bull Available for simultaneous control by a masterbull Size up to 350000 nodes (from todayrsquos paper)

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 90: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

90

ldquoRallyingrdquo the Botnet

bull Easy to combine worm backdoor functionalitybull Problem how to learn about successfully

infected machines

bull Optionsndash Emailndash Hard-coded email address

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 91: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

91

Botnet Control

bull Botnet master typically runs some IRC server on a well-known port (eg 6667)

bull Infected machine contacts botnet with pre-programmed DNS name (eg big-botde)

bull Dynamic DNS allows controller to move about freely

Infected Machine

DynamicDNS

BotnetController

(IRC server)

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 92: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

92

Some Defenses

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 93: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

93

Idea 1 Ingress Filtering

bull RFC 2827 Routers install filters to drop packets from networks that are not downstream

bull Feasible at edgesbull Difficult to configure closer to network ldquocorerdquo

20469207024 Internet

Drop all packets with source address other than 20469207024

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 94: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

94

Idea 2 uRPF Checks

bull Unicast Reverse Path Forwardingndash Cisco ldquoip verify unicast reverse-pathrdquo

bull Requires symmetric routing

Accept packet from interface only if forwarding table entry for source IP address matches ingress interface

100183

A10018

10015

101203

100181 24

10011 24

Strict Mode uRPF

Enabled

ldquoArdquo Routing TableDestination Next Hop1001024 Int 110018024 Int 2

100183 from wrong interface

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 95: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

95

Problems with uRPF

bull Asymmetric routing

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 96: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

96

S-BGP

bull Address-based PKI validate signaturesndash Authentication of

bull ownership for IP address blocks bull AS number bull an ASs identity and bull a BGP routers identity

ndash Use existing infrastructure (Internet registries etc)ndash Routing origination is digitally signedndash BGP updates are digitally signed

1048708 bull Route attestations A new optional BGP transitive path attribute

ndash carries digital signatures covering the routing information in updates

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP
Page 97: Networking Nick Feamster Georgia Tech. 2 Goal of This Tutorial Teach engineers the basics of networking and ISP operations Networks today –Business models.

97

Practical Problems with S-BGP

bull Requires Public-Key Infrastructure

bull Lots of digital signatures to calculate and verifyndash Message overheadndash CPU overhead

bull Calculation expense is greatest when topology is changingndash Caching can help

bull Route aggregation is problematic (maybe thatrsquos OK)

bull Secure route withdrawals when link or node fails

bull Address ownership data out of date

bull Deployment

  • Networking
  • Goal of This Tutorial
  • Todayrsquos Networks
  • Business Models
  • Billing for Internet Usage
  • Net Neutrality
  • Network Operations
  • Point-of-Presence (PoP)
  • Example Abilene Network Topology
  • Another Example Backbone
  • Internet Routing Overview
  • Internet Routing Protocol BGP
  • Two Flavors of BGP
  • IPv4 Addresses Networks of Networks
  • Pre-1994 Classful Addressing
  • Classless Interdomain Routing (CIDR)
  • Benefits of CIDR
  • Growth of IP Prefixes
  • 1994-1998 Linear Growth
  • Around 2000 Fast Growth Resumes
  • Fast growth resumes
  • The Address Allocation Process
  • Common Problems
  • What can go wrong
  • Measurement and Monitoring
  • Passive vs Active Measurement
  • Slide 27
  • Passive Traffic Data Measurement
  • Simple Network Management Protocol
  • SNMP (Passive)
  • Packet-level Monitoring
  • Full Packet Capture (Passive)
  • What is a flow
  • Cisco NetFlow
  • Flow Record Contents
  • Aggregating Packets into Flows
  • Reducing Measurement Overhead
  • Packet Sampling for Flow Monitoring
  • Sampling Flow-Level Sampling
  • Two Main Approaches
  • Packet Capture on High-Speed Links
  • Characteristics of Packet Capture
  • Data Measurement Repositories
  • Multihoming and Traffic Engineering
  • What is Multihoming
  • Why Multihome
  • Redundancy
  • Performance
  • Multihoming in IP Networks Today
  • BGP or no
  • Same Provider or Multiple
  • Multihomed Stub One Link
  • Multihomed Stub Multiple Links
  • Multihomed Stub Multiple ISPs
  • Multihomed Transit Network
  • Interdomain Traffic Engineering
  • Outbound Traffic Control
  • Outbound Traffic Load Balancing
  • Traffic Engineering Goals
  • Managing Scale
  • Achieving Predictability
  • Challenges to Predictability
  • Inter-AS Negotiation
  • Outbound Multihoming Goals
  • Inbound Traffic Control
  • How many links are enough
  • Problems with Multihoming in IPv4
  • Slide 68
  • Slide 69
  • Diagnosis and Troubleshooting
  • Operator Mailing List (NANOG)
  • Operator Mailing List
  • Routing Configuration
  • Internet Business Model (Simplified)
  • Peering Contracts Consistent Export
  • Consistent Export
  • Inconsistent Export in Practice
  • Blackholes
  • Security
  • Security ldquoBogonrdquo Routes
  • Spam Phishing etc
  • Spam and Routing
  • Worms and Botnets
  • What is a Worm
  • Example Worm Code Red
  • Code Red Revisions
  • Code Red Host Infection Rate
  • Designing Fast-Spreading Worms
  • Botnets
  • ldquoRallyingrdquo the Botnet
  • Botnet Control
  • Some Defenses
  • Idea 1 Ingress Filtering
  • Idea 2 uRPF Checks
  • Problems with uRPF
  • S-BGP
  • Practical Problems with S-BGP

Top Related