© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 1
Troubleshooting BGP
Philip Smith <[email protected]>APRICOT 200820-29 February, Taipei, Taiwan
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 2
Presentation Slides
Available onftp://ftp-eng.cisco.com/pfs/seminars/APRICOT2008-Troubleshooting-BGP.pdfAnd on the APRICOT 2008 website
Feel free to ask questions any time
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 3
Assumptions
Presentation assumes working knowledge of BGPBeginner and Intermediate experience of protocol
Knowledge of Cisco CLIHopefully you can translate concepts into your own router CLI
If in any doubt, please ask!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 4
Agenda
Fundamentals of Troubleshooting
Local Configuration Problems
Internet Reachability Problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 5
Fundamentals:Problem Areas
First step is to recognise what usually causes problems
Possible Problem Areas:Misconfiguration
Configuration errors caused by bad documentation,misunderstanding of concepts, poor communication betweencolleagues or departments
Human errorTypos, using wrong commands, accidents, poorly plannedmaintenance activities
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 6
Fundamentals:Problem Areas
More Possible Problem Areas:“feature behaviour”
Or – “it used to do this with Release X.Y(a) but ReleaseX.Y(b) does that”
Interoperability issuesDifferences in interpretation of RFC1771 and itsdevelopments
Those beyond your controlUpstream ISP or peers make a change which has anunforeseen impact on your network
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 7
Fundamentals:Working on Solutions
Next step is to try and fix the problemAnd this is not about diving into network and trying randomcommands on random routers, just to “see what difference thismakes”
The best procedure for “unfamiliar problems” is toStart at one place,Deal with one symptom, and learn more about it
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 8
Fundamentals:Working on Solutions
Remember! Troubleshooting is about:Not panickingCreating a checklistWorking to that checklistStarting at the bottom and working up
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 9
Fundamentals:Checklists
This presentation will have references in the laterstages to checklists
They are the best way to work to a solutionThey are what many NOC staff follow when diagnosing andsolving network problemsIt may seem daft to start with simple tests when the problemlooks complex
But quite often the apparently complex can be solved quiteeasily
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 10
Fundamentals:Tools
Use system and network logs as an aid
Record keeping:Good and detailed system logsLast known good configurationHistory trail of working configurations and all intermediatechangesRecord of commands entered on routers and other networkdevices
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 11
Fundamentals:Tools
Familiarise yourself with the router’s tools:Is logging of the BGP process enabled?
(And is it captured/recorded off the router?)Are you familiar with the BGP debug process and commands (ifavailable)
Check vendor documentation before switching on full BGPdebugging – you might get fewer surprises
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 12
Fundamentals:Tools
Traffic and traffic flow measurement in the networkUnexplained change in traffic levels on an interface, aconnection, a peering,…Correlation of customer feedback on network or connectivityissues…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 13
Agenda
Fundamentals
Local Configuration Problems
Internet Reachability Problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 14
Local Configuration Problems
Peer Establishment
Missing Routes
Inconsistent Route Selection
Loops and Convergence Issues
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 15
Peer Establishment
Routers establish a TCP sessionPort 179 – Permit in interface filtersIP connectivity (route from IGP)
OPEN messages are exchangedPeering addresses must match theTCP sessionLocal AS configuration parameters
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 16
Common Problems
Sessions are not establishedNo IP reachabilityIncorrect configuration
Peers are flappingLayer 2 problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 17
R2#sh run | begin ^router bgp
router bgp 1
bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 1
neighbor 3.3.3.3 remote-as 2
AS 1
AS 2
R1iBGP
eBGP
1.1.1.1 2.2.2.2
3.3.3.3?
?
R2
R3
Peer Establishment:Diagram
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 18
R2#show ip bgp summary
BGP router identifier 2.2.2.2, local AS number 1
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State
1.1.1.1 4 1 0 0 0 0 0 never Active
3.3.3.3 4 2 0 0 0 0 0 never Idle
Peer Establishment:Symptoms
Both peers are having problemsState may change between Active, Idle and Connect
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 19
R2#router bgp 1 neighbor 1.1.1.1 remote-as 1 neighbor 3.3.3.3 remote-as 2
Local AS
eBGP Peer
iBGP Peer
Peer Establishment
Is the Local AS configured correctly?
Is the remote-as assigned correctly?
Verify with your diagram or other documentation!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 20
R2#show tcp brief allTCB Local Address Foreign Address (state)005F2934 *.179 3.3.3.3.* LISTEN0063F3D4 *.179 1.1.1.1.* LISTEN
R2#debug ip tcp transactionsTCP special event debugging is onR2#TCP: sending RST, seq 0, ack 2500483296TCP: sent RST to 4.4.4.4:26385 from 2.2.2.2:179
Peer Establishment:iBGP
Assume that IP connectivity has been checked Check TCP to find out what connections we are accepting
We Are Listening for TCP Connections for Port 179 for theConfigured Peering Addresses Only!
Remote Is Trying to Open the Session from 4.4.4.4 Address…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 21
R2#debug ip bgp BGP debugging is onR2#BGP: 1.1.1.1 open active, local address 4.4.4.5BGP: 1.1.1.1 open failed: Connection refused by remote host
R2#sh ip route 1.1.1.1Routing entry for 1.1.1.1/32 Known via "static", distance 1, metric 0 (connected) * directly connected, via Serial1 Route metric is 0, traffic share count is 1
R2#show ip interface brief | include Serial1Serial1 4.4.4.5 YES manual up up
Peer Establishment:iBGP
What about us?
We are trying to open the session from 4.4.4.5 address…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 22
R2#router bgp 1 neighbor 1.1.1.1 remote-as 1 neighbor 1.1.1.1 update-source Loopback0 neighbor 3.3.3.3 remote-as 2 neighbor 3.3.3.3 update-source Loopback0
Peer Establishment:iBGP
Source address is the outgoing interface towards the destinationbut peering in this case is using loopback interfaces!
Force both routers to source from the correct interface
Use “update-source” to specify the loopback when loopbackpeering
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 23
Peer Establishment:iBGP – Summary
Assume that IP connectivity has been checkedIncluding IGP reachability between peers
Check TCP to find out what connections we are acceptingCheck the ports and source/destination addressesDo they match the configuration?
Common problem:iBGP is run between loopback interfaces on router (for stability), butthe configuration is missing from the router ⇒ iBGP fails to establishRemember that source address is the IP address of the outgoinginterface unless otherwise specified
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 24
AS 1
AS 2
R1iBGP
eBGP
1.1.1.1 2.2.2.2
3.3.3.3
?
R2
R3
Peer Establishment:Diagram
R1 is established now
The eBGP session is still having trouble!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 25
R2#ping 3.3.3.3Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms
Peer Establishment:eBGP
Trying to load-balance over multiple links to the eBGPpeer
Verify IP connectivityCheck the routing tableUse ping/trace to verify two way reachability
Routing towards destination is correct, but…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 26
R2#ping ipTarget IP address: 3.3.3.3Extended commands [n]: ySource address or interface: 2.2.2.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:.....Success rate is 0 percent (0/5)
Peer Establishment:eBGP
Use extended pings to test loopback to loopback connectivity
R3 does not have a route to our loopback, 2.2.2.2
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 27
R2#sh ip bgp neigh 3.3.3.3BGP neighbor is 3.3.3.3, remote AS 2, external link BGP version 4, remote router ID 0.0.0.0 BGP state = Idle Last read 00:00:04, hold time is 180, keepalive interval is 60 seconds Received 0 messages, 0 notifications, 0 in queue Sent 0 messages, 0 notifications, 0 in queue Route refresh request: received 0, sent 0 Default minimum time between advertisement runs is 30 seconds For address family: IPv4 Unicast BGP table version 1, neighbor version 0 Index 2, Offset 0, Mask 0x4 0 accepted prefixes consume 0 bytes Prefix advertised 0, suppressed 0, withdrawn 0 Connections established 0; dropped 0 Last reset never External BGP neighbor not directly connected. No active TCP connection
Peer Establishment:eBGP Assume R3 added a route to 2.2.2.2
Still having problems…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 28
R2#router bgp 1 neighbor 3.3.3.3 remote-as 2 neighbor 3.3.3.3 ebgp-multihop 2 neighbor 3.3.3.3 update-source Loopback0
Peer Establishment:eBGP
eBGP peers are normally directly connectedBy default, TTL is set to 1 for eBGP peersIf not directly connected, specify ebgp-multihop
At this point, the session should come up
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 29
R2#show ip bgp summaryBGP router identifier 2.2.2.2, local AS number 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State3.3.3.3 4 2 10 26 0 0 0 never Active
Peer Establishment:eBGP
Still having trouble!Connectivity issues have already been checked and corrected
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 30
R2#debug ip bgp events14:06:37: BGP: 3.3.3.3 open active, local address 2.2.2.214:06:37: BGP: 3.3.3.3 went from Active to OpenSent14:06:37: BGP: 3.3.3.3 sending OPEN, version 414:06:37: BGP: 3.3.3.3 received NOTIFICATION 2/2
(peer in wrong AS) 2 bytes 000114:06:37: BGP: 3.3.3.3 remote close, state CLOSEWAIT14:06:37: BGP: service reset requests14:06:37: BGP: 3.3.3.3 went from OpenSent to Idle14:06:37: BGP: 3.3.3.3 closing
Peer Establishment:eBGP
If an error is detected, a notification is sent and the sessionis closed
R3 is configured incorrectlyHas “neighbor 2.2.2.2 remote-as 10”Should have “neighbor 2.2.2.2 remote-as 1”
After R3 makes this correction the session should come up
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 31
access-list 100 permit tcp host 3.3.3.3 eq 179 host 2.2.2.2access-list 100 permit tcp host 3.3.3.3 host 2.2.2.2 eq 179
Peer Establishment:eBGP – Summary
Remember to allow TCP/179 through edge filters
Be very careful with multihop eBGPCheck IP connectivity (local and remote routing tables)Remember to source updates from loopbackWatch for filters anywhere in the pathTTL must be at least 2 for ebgp-multihop between directlyconnected neighbours
Use TTL value carefully
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 32
R2#show ip bgp summary BGP router identifier 2.2.2.2, local AS number 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd3.3.3.3 4 2 10 26 0 0 0 never Active
Peer Establishment:Passwords
Using passwords on iBGP and eBGP sessionsLink won’t come upBeen through all the previous troubleshooting steps
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 33
R2#router bgp 1 neighbor 3.3.3.3 remote-as 2 neighbor 3.3.3.3 ebgp-multihop 2 neighbor 3.3.3.3 update-source Loopback0 neighbor 3.3.3.3 password 7 05080F1C221C
%TCP-6-BADAUTH: No MD5 digest from 3.3.3.3:179to 2.2.2.2:11272%TCP-6-BADAUTH: No MD5 digest from 3.3.3.3:179to 2.2.2.2:11272%TCP-6-BADAUTH: No MD5 digest from 3.3.3.3:179to 2.2.2.2:11272
Peer Establishment:Passwords
Configuration on R2 looks fine!
Check the log messages – enable “log-neighbor-changes”
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 34
R3#router bgp 2 neighbor 2.2.2.2 remote-as 1 neighbor 2.2.2.2 ebgp-multihop 2 neighbor 2.2.2.2 update-source Loopback0
Peer Establishment:Passwords
Check configuration on R3Password is missing from the eBGP configuration
Fix the R3 configurationPeering should now come up!But it does not
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 35
Peer Establishment:Passwords
Let’s look at the log messages again for clues
R2#
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11024 to 2.2.2.2:179
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11024 to 2.2.2.2:179
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11024 to 2.2.2.2:179
We are getting invalid MD5 digest messages – passwordmismatch!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 36
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11027to 2.2.2.2:179%BGP-5-ADJCHANGE: neighbor 3.3.3.3 Up
Peer Establishment:Passwords
We must have mis-typed the password on one of thepeering routers
Fix the password – best to re-enter password on both routerseBGP session now comes up
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 37
Peer Establishment:Passwords – Summary
Common problems:Missing password – needs to be on both endsCut and paste errors – don’t!Typographical & transcription errorsCapitalisation, extra characters, white space…
Common solutions:Check for symptoms/messages in the logsRe-enter passwords using keyboard, from scratch – don’tcut&paste
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 38
AS 2AS 1
Layer 2
eBGP R2R1
Flapping Peer:Common Symptoms
Symptoms – the eBGP session flaps
eBGP peering establishes, then drops, re-establishes, thendrops,…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 39
R2#
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent
%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold timeexpired) 0 bytes
R2#show ip bgp neighbor 1.1.1.1 | include Last reset
Last reset 00:01:02, due to BGP Notification sent, hold timeexpired
Flapping Peer
Ensure BGP neighbour logging is enabledno logs ⇒ no clue what is going on
R1 and R2 are peering over some 3rd party L2 network
We are not receiving keepalives from the other side!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 40
R1#show ip bgp summaryBGP router identifier 172.16.175.53, local AS number 1BGP table version is 10167, main routing table version 10167
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 97 00:02:15 0
R1#show ip bgp summary | begin NeighborNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 98 00:03:04 0
Flapping Peer
Let’s take a look at our peer!
Hellos are stuck in OutQ behind update packets!
Notice that the MsgSent counter has not moved
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 41
R1#ping 2.2.2.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 16/21/24 ms
R1#ping ipTarget IP address: 2.2.2.2Repeat count [5]:Datagram size [100]: 1500Timeout in seconds [2]:Extended commands [n]:Sweep range of sizes [n]:Type escape sequence to abort.Sending 5, 1500-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:.....Success rate is 0 percent (0/5)
Flapping Peer
Normal pings work but a 1500byte ping fails?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 42
Flapping Peer:Diagnosis and Solution
DiagnosisKeepalives get lost because they get stuck in the router’s queuebehind BGP update packets.BGP update packets are packed to the size of the MTU – keepalivesand BGP OPEN packets are not packed to the size of the MTU ⇒ PathMTU problemsUse ping with different size packets to confirm the above – 100byteping succeeds, 1500byte ping fails = MTU problem somewhere
SolutionPass the problem to the L2 folks – but be helpful, try and pinpoint usingping where the problem might be in the network
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 43
Flapping Peer:Other Common Problems
Remote router rebooting continually (typical with a 3-5 minute BGPpeering cycle time)
Remote router BGP process unstable, restarting Traffic Shaping & Rate Limiting parameters MTU incorrectly set on links, PMTU discovery disabled on router For non-ATM/FR links, instability in the L2 point-to-point circuits
Faulty MUXes, bad connectors, interoperability problems, PPPproblems, satellite or radio problems, weather, etc. The list is endless –your L2 folks should know how to solve themFor you, ping is the tool to use
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 44
AS 2AS 1
Layer 2
eBGP R2R1
Small Packets
Large Packets
Flapping Peer:Fixed!
Large packets are ok now
BGP session is stable!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 45
Local Configuration Problems
Peer Establishment
Missing Routes
Inconsistent Route Selection
Loops and Convergence Issues
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 46
Quick Review
Once the session has been established, UPDATEs areexchanged
All the locally known routesOnly the bestpath is advertised
Incremental UPDATE messages are exchangedafterwards
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 47
Quick Review
Bestpath received from eBGP peerAdvertise to all peers
Bestpath received from iBGP peerAdvertise only to eBGP peersA full iBGP mesh must exist
(Unless we are using Route Reflectors)
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 48
Missing Routes
Route Origination
UPDATE Exchange
Filtering
iBGP mesh problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 49
Missing Routes:Route Origination
Common problem occurs when putting prefixes into theBGP table
BGP table is NOT the RIB(RIB = Routing Information Base – a.k.a the Routing Table)BGP table, as with OSPF table, ISIS table, static routes, etc, isused to feed the RIB, and hence the FIBEach routing protocol has a different priority or “distance”
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 50
Missing Routes:Route Origination
To get a prefix into BGP, it must exist in another routingprocess too, typically:
Static route pointing to customer (for customer routes into youriBGP)Static route pointing to Null (for aggregates you want to put intoyour eBGP)
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 51
Network statement
BGP is not originating the route???
Do we have the exact route?
R1# show run | include 200.200.0.0
network 200.200.0.0 mask 255.255.252.0
R1# show ip bgp | include 200.200.0.0
R1#
R1# show ip route 200.200.0.0 255.255.252.0
% Network not in table
Route Origination:Example I
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 52
Route Origination:Example I
Nail down routes you want to originate
Check the RIB
BGP originates the route!!
ip route 200.200.0.0 255.255.252.0 Null0 254
R1# show ip route 200.200.0.0 255.255.252.0 200.200.0.0/22 is subnetted, 1 subnetsS 200.200.0.0 [1/0] via Null 0
R1# show ip bgp | include 200.200.0.0
*> 200.200.0.0/22 0.0.0.0 0 32768
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 53
Route Origination:Example II
Trying to originate an aggregate route
The RIB has a component but BGP does not create theaggregate???
aggregate-address 7.7.0.0 255.255.0.0 summary-only
R1# show ip route 7.7.0.0 255.255.0.0 longer 7.0.0.0/32 is subnetted, 1 subnetsC 7.7.7.7 [1/0] is directly connected, Loopback 0
R1# show ip bgp | i 7.7.0.0
R1#
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 54
R1# show ip bgp 7.7.0.0 255.255.0.0 longer
R1#
network 7.7.7.7 mask 255.255.255.255
R1# show ip bgp 7.7.0.0 255.255.0.0 longer
*> 7.7.0.0/16 0.0.0.0 32768 i
s> 7.7.7.7/32 0.0.0.0 0 32768 i
Route Origination:Example II
Remember, to have a BGP aggregate you need a BGPcomponent, not a RIB component
Once BGP has a component route we originate the aggregate
s means this component is suppressed due to the “summary-only”argument
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 55
Troubleshooting Tips
BGP Network statement rulesAlways need an exact route (RIB)
aggregate-address looks in the BGP table,not the RIB
Showing RIB component routes:show ip route x.x.x.x y.y.y.y longer
Showing BGP component routes:show ip bgp x.x.x.x y.y.y.y longer
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 56
Missing Routes
Route Origination
UPDATE Exchange
Filtering
iBGP mesh problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 57
Missing Routes:Update Exchange
Ah, Route Reflectors…Such a nice solution to help scale iBGPBut why do people insist in breaking the rules all the time?!
Common issuesClashing router IDsClashing cluster IDs
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 58
R1
R3
R2
R4
Missing Routes:Example I
Two RR clusters
R1 is a RR for R3
R2 is a RR for R4
R4 is advertising 7.0.0.0/8R2 has the routeR1 and R3 do not
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 59
R2# show ip bgp neighbors 1.1.1.1 advertised-routes BGP table version is 2, local router ID is 2.2.2.2 Network Next Hop Metric LocPrf Weight Path*>i7.0.0.0 4.4.4.4 0 100 0 i
R1# show ip bgp neighbors 2.2.2.2 routes
Total number of prefixes 0
Missing Routes:Example I
First, did R2 advertise the route to R1?
Did R1 receive it?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 60
Time to debug!!
Tell R2 to resend his UPDATEs
R1 shows us something interesting
Cannot accept an update with our Router-ID as theORIGINATOR_ID. Another means of loop detection in BGP
access-list 100 permit ip host 7.0.0.0 host 255.0.0.0
R1# debug ip bgp update 100
R2# clear ip bgp 1.1.1.1 out
*Mar 1 21:50:12.410: BGP(0): 2.2.2.2 rcv UPDATE w/ attr:nexthop 4.4.4.4, origin i, localpref 100, metric 0,originator 100.1.1.1, clusterlist 2.2.2.2, path , community ,extended community
*Mar 1 21:50:12.410: BGP(0): 2.2.2.2 rcv UPDATE about7.0.0.0/8 -- DENIED due to: ORIGINATOR is us;
Missing Routes:Example I
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 61
Missing Routes:Example I – Summary
R1 is not accepting the route when R2 sends it on fromits client, R4
R1 and R4 have the same router ID!If R1 sees its own router ID in the originator attribute in anyreceived prefix, it will reject that prefix
This is how a route reflector attempts to avoid routing loops
SolutionDo NOT set the router ID by hand unless you have a very goodreason to do so and have a very good plan for deploymentRouter-ID is usually calculated automatically by router
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 62
R1#show run | include cluster bgp cluster-id 10R2#show run | include cluster bgp cluster-id 10
R1
R3
R2
R4
Missing Routes:Example II
One RR cluster
R1 and R2 are RRs
R3 and R4 are RRCs
R4 is advertising 7.0.0.0/8R2 has the routeR1 and R3 do not
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 63
R2# show ip bgp neighbors 1.1.1.1 advertised-routes
BGP table version is 2, local router ID is 2.2.2.2
Origin codes: i - IGP, e - EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
*>i7.0.0.0 4.4.4.4 0 100 0 i
R1# show ip bgp neighbor 2.2.2.2 routes
Total number of prefixes 0
Missing Routes:Example II
Same troubleshooting steps as for the previous example!
Did R2 advertise it to R1?
Did R1 receive it?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 64
Time to debug!!
Tell R2 to resend his UPDATEs
R1 shows us something interesting
Remember, all RRCs must peer with all RRs in a cluster; allowsR4 to send the update directly to R1
access-list 100 permit ip host 7.0.0.0 host 255.0.0.0
R1# debug ip bgp update 100
R2# clear ip bgp 1.1.1.1 out
Mar 3 14:28:57.208: BGP(0): 2.2.2.2 rcv UPDATE w/ attr:nexthop 4.4.4.4, origin i, localpref 100, metric 0, originator4.4.4.4, clusterlist 0.0.0.10, path , community , extendedcommunity
Mar 3 14:28:57.208: BGP(0): 2.2.2.2 rcv UPDATE about7.0.0.0/8 -- DENIED due to: reflected from the same cluster;
Missing Routes:Example II
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 65
Missing Routes:Example II – Summary
R1 is not accepting the route when R2 sends it onIf R1 sees its own router ID in the cluster-ID attribute in any receivedprefix, it will reject that prefix
How a route reflector avoids redundant information
ReasonEarly documentation claimed that RRC redundancy should beachieved by dual route reflectors in the same clusterThis is fine and good, but then ALL clients must peer with both RRs,otherwise examples like this will occur
SolutionUse overlapping Route Reflector Clusters for redundancy, stay withdefaults
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 66
Troubleshooting Tips
The list of NLRI you sent a peer:show ip bgp neighbor x.x.x.x advertisedNote: The attribute values shown are taken from the BGP table;attribute modifications by outbound route-maps will not be shown
Display the routes sent to us by neighbour x.x.x.x after processingby our inbound filters:
show ip bgp neighbor x.x.x.x routes
Display the routes sent to us by neighbour x.x.x.x prior toprocessing by our inbound filters
show ip bgp neighbor x.x.x.x receivedCan only use if Soft-Reconfiguration is enabled
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 67
alpha#sh ip bgp neigh 192.168.12.1 routes Network Next Hop Metric LocPrf Weight Path*>i1.0.0.0 192.168.12.1 0 50 0 i*>i222.222.0.0/19 192.168.5.1 200 0 3 4 i
alpha#sh ip bgp neigh 192.168.12.1 received-routes Network Next Hop Metric LocPrf Weight Path* i1.0.0.0 192.168.12.1 0 100 0 i* i169.254.0.0 192.168.5.1 0 100 0 3 i* i222.222.0.0/19 192.168.5.1 100 0 3 4 i
Troubleshooting Tips“soft-reconfiguration”
Ideal for troubleshooting problems with inbound filters andattributes
show ip bgp neighbor x.x.x.x routes
show ip bgp neighbor x.x.x.x received
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 68
Missing Routes
Route Origination
UPDATE Exchange
Filtering
iBGP mesh problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 69
Update Filtering
Type of filtersPrefix filtersAS_PATH filtersCommunity filtersRoute-maps
Applied incoming and/or outgoing
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 70
Missing RoutesUpdate Filters
Determine which filters are applied to the BGP sessionshow ip bgp neighbors x.x.x.xshow run | include neighbor x.x.x.x
Examine the route and pick out the relevant attributesshow ip bgp x.x.x.x
Compare the attributes against the filters
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 71
R1#show ip bgp neigh 2.2.2.2 routes
Total number of prefixes 0
R1 R2
10.0.0.0/810.0.0.0/8 ???
Missing RoutesUpdate Filters
Missing 10.0.0.0/8 in R1 (1.1.1.1)
Not received from R2 (2.2.2.2)
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 72
R2#show ip bgp neigh 1.1.1.1 advertised-routesNetwork Next Hop Metric LocPrf Weight Path
R2#show ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 1660Paths: (1 available, best #1) Not advertised to any peer Local 0.0.0.0 from 0.0.0.0 (2.2.2.2) Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best
Missing RoutesUpdate Filters
R2 originates the route
Does not advertise it to R1
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 73
R2#show run | include neighbor 1.1.1.1 neighbor 1.1.1.1 remote-as 3 neighbor 1.1.1.1 filter-list 1 out
R2#sh ip as-path 1 AS path access list 1 permit ^$
Missing RoutesUpdate Filters
Time to check filters!
^ matches the beginning of a line
$ matches the end of a line
^$ means match any empty AS_PATH
Filter “looks” correct
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 74
R2#show ip bgp filter-list 1
R2#show ip bgp regexp ^$BGP table version is 1661, local router ID is 2.2.2.2Status codes: s suppressed, d damped, h history, * valid, > best, i - internalOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path*> 10.0.0.0 0.0.0.0 0 32768 i
Missing RoutesUpdate Filters
Nothing matches the filter-list???
Re-typing the regexp gives the expected output
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 75
R2#show ip bgp regexp ^$
Nothing matches again! Let’s use the up arrow key to see where the cursor stops
R2#show ip bgp regexp ^$ End of Line Is at the Cursor
Missing RoutesUpdate Filters
Copy and paste the entire regexp line from the configuration
There is a trailing white space at the end It is considered part of the regular expression
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 76
R2#clear ip bgp 1.1.1.1 out
R1#show ip bgp 10.0.0.0 % Network not in table
Missing RoutesUpdate Filters
Force R2 to resend the update after the filter-list correction
Then check R1 to see if it has the route
R1 still does not have the route
Time to check R1’s inbound policy for R2
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 77
R1#show run | include neighbor 2.2.2.2 neighbor 2.2.2.2 remote-as 12 neighbor 2.2.2.2 route-map POLICY inR1#show route-map POLICYroute-map POLICY, permit, sequence 10 Match clauses: ip address (access-lists): 100 101 as-path (as-path filter): 1 Set clauses: Policy routing matches: 0 packets, 0 bytesR1#show access-list 100Extended IP access list 100 permit ip host 10.0.0.0 host 255.255.0.0R1#show access-list 101Extended IP access list 101 permit ip 200.1.0 0.0.0.255 host 255.255.255.0R1#show ip as-path 1AS path access list 1 permit ^12$
Missing RoutesUpdate Filters
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 78
R1#show access-list 99Standard IP access list 99 permit 10.0.0.0
R1#debug ip bgp 2.2.2.2 update 99BGP updates debugging is on for access list 99 for neighbor 2.2.2.2
R1#4d00h: BGP(0): 2.2.2.2 rcvd UPDATE w/ attr: nexthop 2.2.2.2, origin i, metric 0, path 124d00h: BGP(0): 2.2.2.2 rcvd 10.0.0.0/8 -- DENIED due to: route-map;
R1 R2
10.0.0.0/810.0.0.0/8 ???
Missing RoutesUpdate Filters
Confused? Let’s run some debugs
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 79
R1#sh run | include neighbor 2.2.2.2 neighbor 2.2.2.2 remote-as 12 neighbor 2.2.2.2 route-map POLICY inR1#sh route-map POLICYroute-map POLICY, permit, sequence 10 Match clauses: ip address (access-lists): 100 101 as-path (as-path filter): 1 Set clauses: Policy routing matches: 0 packets, 0 bytesR1#sh access-list 100Extended IP access list 100 permit ip host 10.0.0.0 host 255.255.0.0R1#sh access-list 101Extended IP access list 101 permit ip 200.1.1.0 0.0.0.255 host 255.255.255.0R1#sh ip as-path 1AS path access list 1 permit ^12$
Missing RoutesUpdate Filters
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 80
Missing RoutesUpdate Filters
Wrong mask! Needs to be /8 and the ACL allows a /16 only!access list 100permit ip host 10.0.0.0 host 255.255.0.0
Should beaccess list 100permit ip host 10.0.0.0 host 255.0.0.0
Use prefix-list instead, more difficult to make a mistakeip prefix-list my_filter permit 10.0.0.0/8
What about ACL 101?Multiple matches on the same line are ORedMultiple matches on different lines are ANDed
ACL 101 does not matter because ACL 100 matcheswhich satisfies the OR condition
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 81
Update Filtering:Summary
If you suspect a filtering problem, become familiar withthe router tools to find out what BGP filters are applied
Tip: don’t cut and paste!Many filtering errors and diagnosis problems result from cut andpaste buffer problems on the client, the connection, and eventhe router
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 82
Update Filtering:Common Problems
Typos in regular expressionsExtra characters, missing characters, white space, etcIn regular expressions every character matters, so accuracy ishighly important
Typos in prefix filtersWatch the router CLI, and the filter logic – it may not be asobvious as you think, or as simple as the manual makes outWatch netmask confusion, and 255 profusion – easy to muddle255 with 0 and 225!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 83
R1#show ip bgp neigh 2.2.2.2 routes
Total number of prefixes 0
R1 R2
10.0.0.0/810.0.0.0/8 ???
Missing RoutesCommunity Problems
Missing 10.0.0.0/8 in R1 (1.1.1.1)
Not received from R2 (2.2.2.2)
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 84
R2#show ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 1660Paths: (1 available, best #1) Not advertised to any peer Local 0.0.0.0 from 0.0.0.0 (2.2.2.2) Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best
Missing RoutesCommunity Problems
R2 originates the route
But the community is not setWould be displayed in the “show ip bgp” output
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 85
R2#show ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 1660Paths: (1 available, best #1) Not advertised to any peer Local 0.0.0.0 from 0.0.0.0 (2.2.2.2) Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best Community 2:2 1:50
R2#show run | begin bgprouter bgp 2 network 10.0.0.0 route-map set-community...route-map set-community permit 10 set community 2:2 1:50
Missing RoutesCommunity Problems Fix the configuration so community is set
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 86
R1#show ip bgp neigh 2.2.2.2 routes
Total number of prefixes 0
Missing RoutesCommunity Problems
R2 now advertises prefix with community to R1 But R1 still doesn’t see the prefix
R1 insists there is nothing wrong with their configuration
Configuration verified on R2 No filters blocking announcement on R2 So what’s wrong?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 87
R2#show run | begin bgprouter bgp 2 network 10.0.0.0 route-map set-community neighbor 1.1.1.1 remote-as 1 neighbor 1.1.1.1 prefix-list my-agg out neighbor 1.1.1.1 prefix-list their-agg in!ip prefix-list my-agg permit 10.0.0.0/8ip prefix-list their-agg permit 20.0.0.0/8!route-map set-community permit 10 set community 2:2 1:50
Missing RoutesCommunity Problems Check R2 configuration again!
Looks okay - filters okay, route-map okay
But forgotten “neighbor 1.1.1.1 send-community”Cisco IOS does NOT send communities by default
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 88
R1#show run | begin bgprouter bgp 1 neighbor 2.2.2.2 remote-as 2 neighbor 2.2.2.2 route-map R2-in in neighbor 2.2.2.2 route-map R1-out out!ip community-list 1 permit 1:150!route-map R2-in permit 10 match community 1 set local-preference 150
Missing RoutesCommunity Problems
R2 now advertises prefix with community to R1
But R1 still doesn’t see the prefixNothing wrong on R2 now, so turn attention to R1
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 89
Missing RoutesCommunity Problems
Community match on R1 expects 1:150 to be set onprefix
But R2 is sending 1:50Typo or miscommunication between operations?
R2 is also using the route-map to filterIf the prefix does not have community 1:150 set, it is dropped –there is no next step in the route-mapWatch the route-map rules in Cisco IOS – they are basically:
if <match> then <set> and exit route-mapelse if <match> then <set> and exit route-mapelse if <match> then <set> etc…
Blank route-map line means match everything, set nothing
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 90
R1#show ip bgp neigh 2.2.2.2 routes
Network Next Hop Metric LocPrf Weight Path* 10.0.0.0 2.2.2.2 0 0 2 i
Total number of prefixes 1
R1#show run | begin ^route-maproute-map R2-in permit 10 match community 1 set local-preference 150route-map R2-in permit 20
Missing RoutesCommunity Problems Fix configuration on R2 to set community 1:150 on announcements
to R1 Fix configuration on R1 to also permit prefixes not matching the
route-map – troubleshooting is easier with prefix-filters doing thefiltering
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 91
Missing RoutesCommunity Problems
Watch route-mapsRoute-map rules often catch out operators when they are usedfor filteringAbsence of an appropriate match means the prefix will bediscarded
Remember to configure all routers to send BGPcommunities
Include it in your default template for iBGPIt should be iBGP default in a Service Provider Network
Remember that it is required to send communities for eBGP too
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 92
Missing Routes:Common Community Problems
Each router implementation has different defaults forwhen communities are sent
Some don’t send communitiesOthers do for iBGP and not for eBGPOthers do for both iBGP and eBGP peers
Watch how your implementation handles communitiesThere may be implicit filtering rules
Each ISP has different community policiesNever assume that because communities exist that people willuse them, or pay attention to the ones you send
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 93
Missing Routes:General Problems
Make and then Stick to simple policy rules:Most router implementations have particular rules for filtering ofprefixes, AS-paths, and for manipulating BGP attributesTry not to mix these rules
Rules for manipulating attributes can also be used forfiltering prefixes and ASNs
These can be very powerful, but can also become veryconfusing
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 94
Missing Routes
Route Origination
UPDATE Exchange
Filtering
iBGP mesh problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 95
Missing RoutesiBGP Example I
Symptom: prefixes seen across network, but noconnectivity
Prefixes learned from eBGP peer are passed across iBGPmeshBut no connectivity to those prefixes
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 96
AS 1
AS 3
iBGP eBGP
1.1.1.1 2.2.2.2
3.3.3.3
4.4.4.4
A
B
AS 2
eBGP
R2R1
R5
R4R3
10.10.0.0/24
Missing RoutesiBGP Example I
R3 customers can reach AS2
No other customers connected to AS1 orAS3 can reach AS2
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 97
R3#show ip bgpStatus codes: * valid, > best, i - internal, Network Next Hop Metric LocPrf Weight Path*> 3.0.0.0 10.10.10.10 0 2 5 i*> 4.0.0.0 10.10.10.10 0 2 5 i*> 10.10.0.0/24 10.10.10.10 0 2 i*> 10.20.0.0/16 10.10.10.10 0 2 i
R4#show ip bgp Network Next Hop Metric LocPrf Weight Path* i3.0.0.0 10.10.10.10 100 0 2 5 i* i4.0.0.0 10.10.10.10 100 0 2 5 i* i10.10.0.0/24 10.10.10.10 100 0 2 i* i10.20.0.0/16 10.10.10.10 100 0 2 i
Missing RoutesiBGP Example I
Looking at R3
Looking at R4
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 98
Missing Routes:iBGP Example I
Notice that R3 reports the prefixes learned from AS2Paths are valid (*) and best (>)
Notice that R4 reports the prefixes learned from R3Paths are valid (*) and internal (i)But no best pathThis is the clue…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 99
R4#sh ip bgp 10.10.0.0/24BGP routing table entry for 10.10.0.0/24, version 136Paths: (1 available, no best path) Not advertised to any peer 2, (received & used) 10.10.10.10 (inaccessible) from 3.2.1.2 (3.3.3.3) Origin IGP, metric 0, localpref 100, valid, internal
R4#sh ip route 10.10.0.0 255.255.255.0% Network not in table
R4#sh ip route 10.10.10.10% Network not in table
The clues
Missing Routes:iBGP Example I
Look at the BGP table entry:
Look at the Routing Table entry
The next hop?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 100
Missing Routes:iBGP Example I – Diagnosis
R4 does not use the 10.10.0.0/24 destination becausethere is no valid next-hop
Configuration on R3 has:Either no routing information on how to reach the10.10.10.10/30 point to point link
By forgetting to put the link into the IGPOr not excluded external next-hops from the internal network
By forgetting to set itself as the next-hop for all externallylearned prefixes on the iBGP session with R4
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 101
Missing Routes:iBGP Example I – Solution
Make sure that all the BGP NEXT_HOPs are known bythe IGP
(whether OSPF/ISIS, static or connected routes)If NEXT_HOP is also in iBGP, ensure the iBGP distance islonger than the IGP distance
—or—
Don’t carry external NEXT_HOPs in your networkReplace eBGP next_hop with local router address on all theedge BGP routers(Cisco IOS “next-hop-self”)
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 102
R4#show ip bgp Network Next Hop Metric LocPrf Weight Path*>i3.0.0.0 3.3.3.3 100 0 2 5 i*>i4.0.0.0 3.3.3.3 100 0 2 5 i*>i10.10.0.0/24 3.3.3.3 100 0 2 i*>i10.20.0.0/16 3.3.3.3 100 0 2 i
Missing RoutesiBGP Example I – Solution
R3 now includes the missing “next-hop-self”configuration
Looking at R4 now:
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 103
Missing RoutesiBGP Example II
Symptom: customer complains about patchy Internetaccess
Can access some, but not all, sites connected to backboneCan access some, but not all, of the Internet
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 104
AS 1
AS 3
iBGP eBGP
1.1.1.1 2.2.2.2
3.3.3.3
4.4.4.4
A
B
AS 2
eBGP
R2R1
R5
R4R3
10.10.0.0/24
Missing RoutesiBGP Example II
Customer connected to R1 can see AS3,but not AS2
Also complains about not being able tosee sites connected to R5
No complaints from other customers
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 105
Missing RoutesiBGP Example II
Diagnosis: This is the classic iBGP mesh problemThe full mesh isn’t complete – how do we know this?
Customer is connected to R1Can’t see AS2 ⇒ R3 is somehow not passing routinginformation about AS2 to R1Can’t see R5 ⇒ R5 is somehow not passing routing informationabout sites connected to R5But can see rest of the Internet ⇒ his prefix is being announcedto some places, so not an iBGP origination problem
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 106
R3#sh ip bgp sum | begin ^NeighNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 200 20 32 0 0 3d10h Active2.2.2.2 4 1 210 25 32 0 0 3d16h 154.4.4.4 4 1 213 22 32 0 0 3d16h 125.5.5.5 4 1 215 19 32 0 0 3d16h 010.10.10.10 4 2 2501 2503 32 0 0 3d16h 100R3#
Missing RoutesiBGP Example II
BGP summary shows that the peering with router R1 isdown
Up/Down is 3 days 10 hours, yet activeWhich means it was last up 3 days and 10 hours agoSo something has broken between R1 and R3
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 107
R1#sh conf | b bgprouter bgp 1 neighbor iBGP-ipv4-peers peer-group neighbor iBGP-ipv4-peers remote-as 1 neighbor iBGP-ipv4-peers update-source Loopback0 neighbor iBGP-ipv4-peers send-community neighbor iBGP-ipv4-peers prefix-list ibgp-prefixes out neighbor 2.2.2.2 peer-group iBGP-ipv4-peers neighbor 4.4.4.4 peer-group iBGP-ipv4-peers neighbor 5.5.5.5 peer-group iBGP-ipv4-peers
Missing RoutesiBGP Example II
Now check configuration on R1
Where is the peering with R3? Restore the missing line, and the iBGP with R3 comes
back up
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 108
R3#sh ip bgp sum | begin ^NeighNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 200 20 32 0 0 00:00:50 82.2.2.2 4 1 210 25 32 0 0 3d16h 154.4.4.4 4 1 213 22 32 0 0 3d16h 125.5.5.5 4 1 215 19 32 0 0 3d16h 010.10.10.10 4 2 2501 2503 32 0 0 3d16h 100R3#
Missing RoutesiBGP Example II
BGP summary shows that no prefixes are being heardfrom R5
This could be due to inbound filters on R3 on the iBGP with R5But there were no filters in the configuration on R3
This must be due to outbound filters on R5 on the iBGP with R3
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 109
R5#sh conf | b neighbor 3.3.3.3 neighbor 3.3.3.3 remote-as 1 neighbor 3.3.3.3 update-source loopback0 neighbor 3.3.3.3 prefix-list ebgp-filters out neighbor 4.4.4.4 remote-as 1 neighbor 4.4.4.4 update-source loopback0 neighbor 4.4.4.4 prefix-list ibgp-filters out!ip prefix-list ebgp-filters permit 20.0.0.0/8ip prefix-list ibgp-filters permit 10.0.0.0/8
Missing RoutesiBGP Example II
Now check configuration on R5
Error in prefix-list in R3 iBGP peeringebgp-filters has been used instead of ibgp-filtersTypo — another advantage of using peer-groups!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 110
R3#sh ip bgp sum | begin ^NeighNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 200 20 32 0 0 00:01:53 82.2.2.2 4 1 210 25 32 0 0 3d16h 154.4.4.4 4 1 213 22 32 0 0 3d16h 125.5.5.5 4 1 215 19 32 0 0 3d16h 610.10.10.10 4 2 2501 2503 32 0 0 3d16h 100R3#
Missing RoutesiBGP Example II
Fix the prefix-list on R5
Check the iBGP again on R3Peering with R1 is upPeering with R5 has prefixes
Confirm that all is okay with customer
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 111
Troubleshooting Tips
Watch the iBGP full meshUse peer-groups both for efficiency and to avoid making policyerrors within the iBGP meshUse route reflectors to avoid accidentally missing iBGP peers,especially as the mesh grows in size
Watch the next-hop for external paths
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 112
Local Configuration Problems
Peer Establishment
Missing Routes
Inconsistent Route Selection
Loops and Convergence Issues
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 113
Inconsistent Route Selection
Two common problems with route selectionInconsistencyAppearance of an incorrect decision
RFC 1771 defined the decision algorithm
Every vendor has tweaked the algorithmhttp://www.cisco.com/warp/public/459/25.shtml
Route selection problems can result from oversights by RFC 1771
RFC1771 is now made obsolete by RFC4271Hopefully compliance with RFC4271 will help avoid future issues
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 114
Inconsistent Route Selection:Example I
RFC1771 said that MED is not always compared
As a result, the ordering of the paths can effect thedecision process
For example, the default in Cisco IOS is to compare theprefixes in order of arrival (most recent to oldest)
This can result in inconsistent route selectionSymptom is that the best path chosen after each BGP reset isdifferent
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 115
Inconsistent Route Selection:Example I
Inconsistent route selection may cause problemsRouting loopsConvergence loops—i.e. the protocol continuously sendsupdates in an attempt to convergeChanges in traffic patterns
Difficult to catch and troubleshootIn Cisco IOS, the deterministic-med configurationcommand is used to order paths consistently
Recommend enabling on all the routers in the ASThe bestpath is recalculated as soon as the commandis entered
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 116
AS 3
AS 2
AS 1
RouterA
AS 1010.0.0.0/8
MED 20MED 30
MED 0
R2R3
R1
Symptom I:Diagram
RouterA will have three paths MEDs from AS 3 will not be compared with
MEDs from AS 1 RouterA will sometimes select the path from R1 as best and may also
select the path from R3 as best
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 117
RouterA#sh ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 40Paths: (3 available, best #3, advertised over iBGP, eBGP) 3 10 2.2.2.2 from 2.2.2.2 Origin IGP, metric 20, localpref 100, valid, internal 3 10 3.3.3.3 from 3.3.3.3 Origin IGP, metric 30, valid, external 1 10 1.1.1.1 from 1.1.1.1 Origin IGP, metric 0, localpref 100, valid, internal, best
Inconsistent Route Selection:Example I
Initial StatePath 1 beats Path 2 – Lower MEDPath 3 beats Path 1 – Lower Router-ID
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 118
RouterA#sh ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 40Paths: (3 available, best #3, advertised over iBGP, eBGP) 1 10 1.1.1.1 from 1.1.1.1 Origin IGP, metric 0, localpref 100, valid, internal 3 10 2.2.2.2 from 2.2.2.2 Origin IGP, metric 20, localpref 100, valid, internal 3 10 3.3.3.3 from 3.3.3.3 Origin IGP, metric 30, valid, external, best
Inconsistent Route Selection:Example I
1.1.1.1 bounced so the paths are re-orderedPath 1 beats Path 2 – Lower Router-IDPath 3 beats Path 1 – External vs Internal
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 119
Deterministic MED:Operation
The paths are ordered by Neighbour AS
The bestpath for each Neighbour AS group is selected
The overall bestpath results from comparing thewinners from each group
The bestpath will be consistent because paths will beplaced in a deterministic order
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 120
RouterA#sh ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 40Paths: (3 available, best #1, advertised over iBGP, eBGP) 1 10 1.1.1.1 from 1.1.1.1 Origin IGP, metric 0, localpref 100, valid, internal, best 3 10 2.2.2.2 from 2.2.2.2 Origin IGP, metric 20, localpref 100, valid, internal 3 10 3.3.3.3 from 3.3.3.3 Origin IGP, metric 30, valid, external
Deterministic MED:Result
Path 1 is best for AS 1
Path 2 beats Path 3 for AS 3 – Lower MED
Path 1 beats Path 2 – Lower Router-ID
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 121
Deterministic MED:Summary
Always use “bgp deterministic-med”
Need to enable throughout entire network at roughlythe same time
If only enabled on a portion of the network routing loopsand/or convergence problems may become moresevere
As a result, default behaviour cannot be changed sothe knob must be configured by the user
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 122
AS 3
AS 2
AS 1
RouterA
AS 1010.0.0.0/8
MED 20MED 30
MED 0
R2R3
R1
Inconsistent Route Selection:Solution – Diagram
RouterA will have three paths
RouterA will consistently select the path from R1 as best!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 123
R3#show ip bgp 7.0.0.0BGP routing table entry for 7.0.0.0/8, version 15 10 100 1.1.1.1 from 1.1.1.1 Origin IGP, metric 0, localpref 100, valid, external 20 100 2.2.2.2 from 2.2.2.2 Origin IGP, metric 0, localpref 100, valid, external, best
R3
AS 10 AS 20
R1 R2
Inconsistent Route Selection:Example II
The bestpath changes everytime the peering is reset
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 124
R3#show ip bgp 7.0.0.0BGP routing table entry for 7.0.0.0/8, version 17Paths: (2 available, best #2) Not advertised to any peer 20 100 2.2.2.2 from 2.2.2.2 Origin IGP, metric 0, localpref 100, valid, external 10 100 1.1.1.1 from 1.1.1.1 Origin IGP, metric 0, localpref 100, valid, external, best
Inconsistent Route Selection:Example II
The “oldest” external is the bestpathAll other attributes are the sameStability enhancement!!—CSCdk12061—Integrated in 12.0(1)
“bgp bestpath compare-router-id” will disable thisenhancement—CSCdr47086—Integrated in 12.0(11)S and 12.1(3)
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 125
R1#sh ip bgp 11.0.0.0BGP routing table entry for 11.0.0.0/8, version 10 100 1.1.1.1 from 1.1.1.1 Origin IGP, localpref 120, valid, internal 100 2.2.2.2 from 2.2.2.2 Origin IGP, metric 0, localpref 100, valid, external, best
Inconsistent Route Selection:Example III
Path 1 has higher localpref but path 2 is better???
This appears to be incorrect…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 126
Path is from an internal peer which means the path must besynchronized by default
Check to see if synchronization is on or off
Sync is still enabled, check for IGP path:
CSCdr90728 “BGP: Paths are not marked as notsynchronized”—Fixed in 12.1(4)
Path 1 is not synchronized Router made the correct choice
Inconsistent Route Selection:Example III
R1# show run | include syncR1#
R1# show ip route 11.0.0.0% Network not in table
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 127
Inconsistent Path Selection
Summary:RFC1771 wasn’t prefect when it came to path selection – yearsof operational experience have shown thisVendors and ISPs have worked to put in stabilityenhancements, now reflected in RFC4271But these can lead to interesting problemsAnd of course some defaults linger much longer than they oughtto – so never assume that an out of the box defaultconfiguration will be perfect for your network
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 128
Local Configuration Problems
Peer Establishment
Missing Routes
Inconsistent Route Selection
Loops and Convergence Issues
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 129
Route Oscillation
One of the most common problems
Main symptom is that traffic exiting the networkoscillates every minute between two exit points
This is almost always caused by the BGP NEXT_HOP beingknown only by BGPCommon problem in ISP networks – but if you have never seenit before, it can be a nightmare to debug and fix
Other symptom is high CPU utilisation for the BGProuter process
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 130
AS 3
AS 12AS 4
R1
R2
R3
142.108.10.2
Route Oscillation:Diagram
R3 prefers routes via AS 4 one minute
BGP scanner runs then R3 prefers routes via AS 12
The entire table oscillates every 60 seconds
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 131
R3#show ip bgp summaryBGP router identifier 3.3.3.3, local AS number 3BGP table version is 502, main routing table version 502267 network entries and 272 paths using 34623 bytes of memory
R3#sh ip route summary | begin bgpbgp 3 4 6 520 1400 External: 0 Internal: 10 Local: 0internal 5 5800Total 10 263 13936 43320
Route Oscillation:Diagnosis
Watch for:Table version number incrementing rapidlyNumber of networks/paths or external/internalroutes changing
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 132
R3#show ip route 156.1.0.0Routing entry for 156.1.0.0/16 Known via "bgp 3", distance 200, metric 0 Routing Descriptor Blocks: * 1.1.1.1, from 1.1.1.1, 00:00:53 ago Route metric is 0, traffic share count is 1 AS Hops 2, BGP network version 474
R3#show ip bgp 156.1.0.0BGP routing table entry for 156.1.0.0/16, version 474Paths: (2 available, best #1) Advertised to non peer-group peers: 2.2.2.2 4 12 1.1.1.1 from 1.1.1.1 (1.1.1.1) Origin IGP, localpref 100, valid, internal, best 12 142.108.10.2 (inaccessible) from 2.2.2.2 (2.2.2.2) Origin IGP, metric 0, localpref 100, valid, internal
Route Oscillation:Troubleshooting
Pick a route from the RIB that has changed within the last minute
Monitor that route to see if it changes every minute
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 133
R3#sh ip route 156.1.0.0Routing entry for 156.1.0.0/16 Known via "bgp 3", distance 200, metric 0 Routing Descriptor Blocks: * 142.108.10.2, from 2.2.2.2, 00:00:27 ago Route metric is 0, traffic share count is 1 AS Hops 1, BGP network version 478
R3#sh ip bgp 156.1.0.0BGP routing table entry for 156.1.0.0/16, version 478Paths: (2 available, best #2) Advertised to non peer-group peers: 1.1.1.1 4 12 1.1.1.1 from 1.1.1.1 (1.1.1.1) Origin IGP, localpref 100, valid, internal 12 142.108.10.2 from 2.2.2.2 (2.2.2.2) Origin IGP, metric 0, localpref 100, valid, internal, best
Route Oscillation:Troubleshooting Check again after bgp_scanner runs
bgp_scanner runs every 60 seconds and validates reachability toall nexthops
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 134
R3#show ip route 142.108.10.2Routing entry for 142.108.0.0/16 Known via "bgp 3", distance 200, metric 0 Routing Descriptor Blocks: * 142.108.10.2, from 2.2.2.2, 00:00:50 ago Route metric is 0, traffic share count is 1 AS Hops 1, BGP network version 476
R3#show ip bgp 142.108.10.2BGP routing table entry for 142.108.0.0/16, version 476Paths: (2 available, best #2) Advertised to non peer-group peers: 1.1.1.1 4 12 1.1.1.1 from 1.1.1.1 (1.1.1.1) Origin IGP, localpref 100, valid, internal 12 142.108.10.2 from 2.2.2.2 (2.2.2.2) Origin IGP, metric 0, localpref 100, valid, internal, best
Route Oscillation:Troubleshooting
Lets take a closer look at the nexthop
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 135
R3#sh debug BGP events debugging is on BGP updates debugging is on IP routing debugging is onR3#BGP: scanning routing tablesBGP: nettable_walker 142.108.0.0/16 calling revise_routeRT: del 142.108.0.0 via 142.108.10.2, bgp metric [200/0]BGP: revise route installing 142.108.0.0/16 -> 1.1.1.1RT: add 142.108.0.0/16 via 1.1.1.1, bgp metric [200/0]RT: del 156.1.0.0 via 142.108.10.2, bgp metric [200/0]BGP: revise route installing 156.1.0.0/16 -> 1.1.1.1RT: add 156.1.0.0/16 via 1.1.1.1, bgp metric [200/0]
Route Oscillation:Troubleshooting
BGP nexthop is known via BGP
Illegal recursive lookup
Scanner will notice and install the other path in the RIB
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 136
R3#BGP: scanning routing tablesBGP: ip nettable_walker 142.108.0.0/16 calling revise_routeRT: del 142.108.0.0 via 1.1.1.1, bgp metric [200/0]BGP: revise route installing 142.108.0.0/16 -> 142.108.10.2RT: add 142.108.0.0/16 via 142.108.10.2, bgp metric [200/0]BGP: nettable_walker 156.1.0.0/16 calling revise_routeRT: del 156.1.0.0 via 1.1.1.1, bgp metric [200/0]BGP: revise route installing 156.1.0.0/16 -> 142.108.10.2RT: add 156.1.0.0/16 via 142.108.10.2, bgp metric [200/0]
Route Oscillation:Troubleshooting
Route to the nexthop is now valid
Scanner will detect this and re-install the other path
Routes will oscillate forever
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 137
AS 3
AS 12AS 4
R1
R2
R3
142.108.10.2
Route Oscillation:Step by Step
R3 naturally prefers routes from AS 12
R3 does not have an IGP route to 142.108.10.2 which is the next-hop forroutes learned via AS 12
R3 learns 142.108.0.0/16 via AS 4 so 142.108.10.2 becomes reachable
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 138
Route Oscillation:Step by Step
R3 then prefers the AS 12 route for 142.108.0.0/16whose next-hop is 142.108.10.2
This is an illegal recursive lookup
BGP detects the problem when scanner runs and flags142.108.10.2 as inaccessible
Routes through AS 4 are now preferred
The cycle continues forever…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 139
Route Oscillation:Solution
Make sure that all the BGP NEXT_HOPs are known bythe IGP
(whether OSPF/ISIS, static or connected routes)If NEXT_HOP is also in iBGP, ensure the iBGP distance islonger than the IGP distance
—or—
Don’t carry external NEXT_HOPs in your networkReplace eBGP next_hop with local router address on all theedge BGP routers(Cisco IOS “next-hop-self”)
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 140
AS 3
AS 12AS 4
R1
R2
R3
142.108.10.2
Route Oscillation:Solution
R3 now has IGP route to AS 12 next-hop or R2 isusing next-hop-self
R3 now prefers routes via AS 12 all the time No more oscillation!!
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 141
Troubleshooting Tips
High CPU utilisation in the BGP process is normally asign of a convergence problem
Find a prefix that changes every minute
Troubleshoot/debug that one prefix
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 142
Troubleshooting Tips
BGP routing loop?First, check for IGP routing loops to the BGP NEXT_HOPs
BGP loops are normally caused byNot following physical topology in RR environmentMultipath with confederationsLack of a full iBGP mesh
Get the following from each router in the loop pathThe routing table entryThe BGP table entryThe route to the NEXT_HOP
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 143
Convergence Problems
Route reflector with 250route reflector clients
100k routes
BGP will notconverge
RR
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 144
RR# show ip bgp summaryNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd20.3.1.160 4 100 10 5416 9419 0 0 00:00:12 Closing20.3.1.161 4 100 11 4418 8055 0 335 00:10:34 020.3.1.162 4 100 12 4718 8759 0 128 00:10:34 020.3.1.163 4 100 9 3517 0 1 0 00:00:53 Connect20.3.1.164 4 100 13 4789 8759 0 374 00:10:37 020.3.1.165 4 100 13 3126 0 0 161 00:10:37 020.3.1.166 4 100 9 5019 9645 0 0 00:00:13 Closing20.3.1.167 4 100 9 6209 9218 0 350 00:10:38 0
RR#show log | i BGP*May 3 15:27:16: %BGP-5-ADJCHANGE: neighbor 20.3.1.118 Down— BGP Notification sent*May 3 15:27:16: %BGP-3-NOTIFICATION: sent to neighbor 20.3.1.118 4/0 (hold time expired) 0 byt*May 3 15:28:10: %BGP-5-ADJCHANGE: neighbor 20.3.1.52 Down— BGP Notification sent*May 3 15:28:10: %BGP-3-NOTIFICATION: sent to neighbor 20.3.1.52 4/0 (hold time expired) 0 byte
Convergence Problems
Have been trying to converge for 10 minutes Peers keep dropping so we never converge?
Check the log to find out why
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 145
We are either missing hellos or our peers are not sending them
Check for interface input drops
72k drops will definitely cause a few peers to go down
We are missing hellos because the interface input queue is verysmall
A rush of TCP Acks from 250 peers can fill 75 spots in a hurry
Increase the size of the queue
RR# show interface gig 2/0 | include dropsOutput queue 0/40, 0 drops; input queue 0/75, 72390 dropsRR#
RR# show run interface gig 2/0interface GigabitEthernet 2/0 ip address 7.7.7.156 255.255.255.0 hold-queue 2000 in
Convergence Problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 146
Let’s start over and give BGP another chance
No more interface input drops
Our peers are stable!!
RR# show log | include BGPRR#
RR# show interface gig 2/0 | include input dropsOutput queue 0/40, 0 drops; input queue 0/2000, 0 dropsRR#
RR# clear ip bgp *RR#
Convergence Problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 147
Convergence Problems
BGP converged in 25 minutes
Still seems like a long time
What was TCP doing?RR#show tcp stat | begin Sent:Sent: 1666865 Total, 0 urgent packets 763 control packets (including 5 retransmitted) 1614856 data packets (818818410 bytes) 39992 data packets (13532829 bytes) retransmitted 6548 ack only packets (3245 delayed) 1 window probe packets, 2641 window update packets
RR#show ip bgp neighbor | include max data segmentDatagrams (max data segment is 536 bytes):
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 148
1.6 Million packets is high
536 is the default MSS (max segment size) for a TCP connection
Very small considering the amount of data we need to transfer
Enable path mtu discovery
Sets MSS to max possible value
RR#show ip bgp neighbor | include max data segmentDatagrams (max data segment is 536 bytes):Datagrams (max data segment is 536 bytes):
RR#show run | include tcpip tcp path-mtu-discoveryRR#
Convergence Problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 149
RR# clear ip bgp *RR#
RR#show ip bgp neighbor | include max data segmentDatagrams (max data segment is 1460 bytes):Datagrams (max data segment is 1460 bytes):
Convergence Problems
Restart the test one more time
MSS looks a lot better
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 150
TCP sent 1 million fewer packets Path MTU discovery helps reduce overhead by sending more data
per packet
BGP converged in 15 minutes! More respectable time for 250 peers and 100k routes
RR# show tcp stat | begin Sent:Sent: 615415 Total, 0 urgent packets 0 control packets (including 0 retransmitted) 602587 data packets (818797102 bytes) 9609 data packets (7053551 bytes) retransmitted 2603 ack only packets (1757 delayed) 0 window probe packets, 355 window update packets
Convergence Problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 151
Summary/Tips
Use ACLs when enabling debug commands
Ensure that BGP logging is switched on
Ensure that deterministic MED’s are enabled
If the entire table is having problem pick one prefix andtroubleshoot it
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 152
Agenda
Fundamentals
Local Configuration Problems
Internet Reachability Problems
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 153
Internet Reachability Problems
BGP Attribute ConfusionTo Control Traffic in → Send MEDs and AS-PATH prepends onoutbound announcementsTo Control Traffic out → Attach local-preference to inboundannouncements
Troubleshooting of multihoming and transit is oftenhampered because the relationship between routinginformation flow and traffic flow is forgotten
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 154
Internet Reachability ProblemsBGP Path Selection Process
Each vendor has “tweaked” the path selection processKnow it for your router equipment – saves time laterEspecially applies with networks with more than one BGPimplementation presentBest policy is to use supplied “knobs” to ensure consistency –and avoid steps in the process which can lead to inconsistency
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 155
Internet Reachability ProblemsMED Confusion
Default MED on Cisco IOS is ZEROIt may not be this on your router, or your peer’s router
Best not to rely on MEDs for multihoming on multiplelinks to upstream
Their default might be 232-1 resulting in your hoped for best pathbeing their worst path“Workaround”, i.e. current good practice, is to use communitiesrather than MEDs
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 156
Internet Reachability ProblemsCommunity Confusion I
set community in a route-map does just that – itoverwrites any other community set on the prefix
Use additive keyword to add community to existing list
Use Internet format for community (AS:xx) not the 32-bit IETF format
32-bit format is harder for humans to comprehendWhereas AS:xx format is more intuitive/recognisable
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 157
Internet Reachability ProblemsCommunity Confusion II
Cisco IOS never sends community by defaultSome implementations send community by default for iBGPpeeringsSome implementations also send community by default foreBGP peerings
Never assume that your neighbouring AS will honouryour no-export community – ask first!
If you leak iBGP prefixes to your upstream for loadsharingpurposes, this could result in your iBGP prefixes leaking to theInternet
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 158
Internet Reachability ProblemsAS-PATH prepending
20 prepends will not lessen the priority of your path anymore than 10 prepends will – check it out at a LookingGlass
The Internet is on average only 5 ASes deep, maximum ASprepend most ISPs have to use is around this tooKnow you BGP path selection algorithm
Some ISPs limit AS-path lengthsFor example, to drop prefixes with AS-paths longer than 15ASNs:bgp maxas-limit 15
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 159
Internet Reachability ProblemsPrivate ASNs
Private ASes should not ever appear in the Internet
Cisco IOS remove-private-AS command does notremove every instance of a private AS
e.g. won’t remove private AS appearing in the middle of a pathsurrounded by public ASNswww.cisco.com/warp/public/459/32.html
Apparent non-removal of private-ASNs may not be abug, but a configuration error somewhere else
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 160
AS 3AS 1
R3R1
R2
AS 2
192.168.1.0/24
Troubleshooting ConnectivityExample I
Symptom: AS1 announces 192.168.1.0/24 to AS2 but AS3 cannotsee the network
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 161
We are checking eBGP filters on R1 and R2.Remember that R2 access will require cooperationand assistance from your peer
We are checking iBGP across AS2’s network(unneeded step in this case, but usually the nextconsideration). Quite often iBGP is misconfigured, lackof full mesh, problems with RRs, etc.
Troubleshooting ConnectivityExample I
Checklist:AS1 announces, but does AS2 see it?
Does AS2 see it over entire network?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 162
We are checking eBGP configuration on R2. There may be aconfiguration error with as-path filters, or prefix-lists, orcommunities such that only local prefixes get out
We are checking eBGP configuration on R3. Maybe AS3 doesnot know to expect prefixes from AS1 in the peering with AS2,or maybe it has similar errors in as-path or prefix or communityfilters
Troubleshooting ConnectivityExample I
Checklist:Does AS2 send it to AS3?
Does AS3 see all of AS2’s originated prefixes?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 163
Troubleshooting ConnectivityExample I
Troubleshooting connectivity beyond immediate peersis much harder
Relies on your peer to assist you – they have the relationshipwith their BGP peers, not youQuite often connectivity problems are due to the privatebusiness relationship between the two neighbouring ASNs
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 164
AS 3AS 1
R3R1
203.51.206.0
The Internet
Troubleshooting ConnectivityExample II
Symptom: AS1 announces 202.173.147.0/24 to its upstreams butAS3 cannot see the network
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 165
We are checking eBGP filters on R1 and upstreams.Remember that upstreams will need to be able to helpyou with this
We are checking if the upstreams are announcingthe network to anywhere on the Internet. See nextslides on how to do this.
Troubleshooting ConnectivityExample II
Checklist:AS1 announces, but do its upstreams see it?
Is the prefix visible anywhere on the Internet?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 166
Troubleshooting ConnectivityExample II
Help is at hand – the Looking Glass
Many networks around the globe run Looking GlassesThese let you see the BGP table and often run simple ping ortraceroutes from their siteswww.traceroute.org and www.bgp4.as/looking-glasses
Some ISPs, especially those with large and diverse networks, runtheir own internal Looking Glass to aid internal troubleshooting
Next slides have some examples of a typical looking glass inaction
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 167
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 168
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 169
Troubleshooting ConnectivityExample II
Hmmm….
Looking Glass can see 202.173.144.0/21This includes 202.173.147.0/24So the problem must be with AS3, or AS3’s upstream
A traceroute confirms the connectivity
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 170
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 171
Troubleshooting ConnectivityExample II
Help is at hand – RouteViews
The main RouteViews router has BGP feeds fromaround 60 peers
www.routeviews.org explains the projectGives access to a real router, and allows any provider to findout how their prefixes are seen in various parts of the InternetComplements the Looking Glass facilities
Anyway, back to our problem…
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 172
We are checking eBGP configuration on AS3’s upstream.There may be a configuration error with as-path filters, orprefix-lists, or communities such that only local prefixes getout. This needs AS3’s assistance
We are checking eBGP configuration on R3. Maybe AS3does not know to expect the prefix from AS1 in the peeringwith its upstream, or maybe it has some errors in as-path orprefix or community filters
Troubleshooting ConnectivityExample II
Checklist:Does AS3’s upstream send it to AS3?
Does AS3 see any of AS1’s originated prefixes?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 173
Troubleshooting ConnectivityExample II
Troubleshooting across the Internet is harderBut tools are available
Looking Glasses, offering traceroute, ping and BGPstatus are available all over the globe
Most connectivity problems seem to be found at the edge of thenetwork, rarely in the transit coreProblems with the transit core are usually intermittent and shortterm in nature
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 174
AS 3AS 2
R2
The Internet
R1
AS 1
R3
Troubleshooting ConnectivityExample III
Symptom: AS1 is trying to loadshare between its upstreams, buthas trouble getting traffic through the AS2 link
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 175
Troubleshooting ConnectivityExample III
Checklist:What does “trouble” mean?
Is outbound traffic loadsharing okay?Can usually fix this with selectively rejecting prefixes, and usinglocal preferenceGenerally easy to fix, local problem, simple application of policy
Is inbound traffic loadsharing okay?Bigger problem if not…Need to do some troubleshooting if configuration withcommunities, AS-PATH prepends, MEDs and selective leakingof subprefixes don’t seem to help
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 176
We are checking eBGP filters on R1 and R2.Remember that R2 access will require cooperationand assistance from your peer
We are checking iBGP across AS2’s network. Quiteoften iBGP is misconfigured, lack of full mesh,problems with RRs, etc.
Troubleshooting ConnectivityExample III
Checklist:AS1 announces, but does AS2 see it?
Does AS2 see it over entire network?
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 177
Troubleshooting ConnectivityExample III
Checklist:Does AS2 send it to its upstream?
Does the Internet see all of AS2’s originated prefixes?
We are checking eBGP configuration on R2. There maybe a configuration error with as-path filters, or prefix-lists,or communities such that only local prefixes get out
We are checking eBGP configuration on other Internetrouters. This means using looking glasses. And trying tofind one as close to AS2 as possible.
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 178
Troubleshooting ConnectivityExample III
Checklist:Repeat all of the above for AS3
Stopping here and resorting to a huge prepend towardsAS3 won’t solve the problem
There are many common problems – listed on nextslide
And tools to help decipher the problem
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 179
Troubleshooting ConnectivityExample III
No inbound traffic from AS2AS2 is not seeing AS1’s prefix, or is blocking it in inbound filters
A trickle of inbound trafficSwitch on NetFlow (if the router has it) and check the origin ofthe trafficIf it is just from AS2’s network blocks, then is AS2 announcingthe prefix to its upstreams?If they claim they are, ask them to ask their upstream for a BGPRIB dump showing the relevant prefixes – or use a LookingGlass to check
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 180
Troubleshooting ConnectivityExample III
A light flow of traffic from AS2, but 50% less than fromAS3
Looking Glass comes to the rescueLG will let you see what AS2, or AS2’s upstreams areannouncingAS1 may choose this as primary path, but AS2 relationshipwith their upstream may decide otherwise
NetFlow comes to the rescueAllows AS1 to see what the origins are, and with the LG,helps AS1 to find where the prefix filtering culprit might be
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 181
AS 3AS 2
R2
The Internet
R1
AS 1
R3
Troubleshooting ConnectivityExample IV
Symptom: AS1 is loadsharing between its upstreams, but thetraffic load swings randomly between AS2 and AS3
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 182
Checklist:Assume AS1 has done everything in this tutorial so far
L2 problem? Route Flap Damping?
All the configurations look fine, the Looking Glassoutputs look fine, life is wonderful… Apart from thoseannoying traffic swings every hour or so
Since BGP is configured fine, and the net has beenstable for so long, can only be an L2 problem, orRoute Flap Damping side-effect
Troubleshooting ConnectivityExample IV
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 183
Troubleshooting ConnectivityExample IV
L2 – upstream somewhere has poor connectivitybetween themselves and the rest of the Internet
Only real solution is to impress upon upstream that this isn’tgood enough, and get them to fix itOr change upstreams
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 184
Troubleshooting ConnectivityExample IV
Route Flap DampingRIPE-378 describes impact of route flap damping on Internet
www.ripe.net/docs/ripe-378.htmlStrongly discouraged in its current form
Many ISPs still implement route flap dampingMany ISPs simply use the vendor defaults
Vendor defaults are too severe
Again Looking Glasses come to the operator’sassistance
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 185
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 186
Troubleshooting ConnectivityExample IV
Several Looking Glasses allow the operators to checkthe flap or damped status of their announcements
Many oscillating connectivity issues are usually caused by L2problemsRoute flap damping will cause connectivity to persist viaalternative paths even though primary paths have been restoredQuite often, the exponential back off of the flap damping timerwill give rise to bizarre routing
Common symptom is that bizarre routing will often clearaway by itself
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 187
Troubleshooting Summary
Most troubleshooting is about:
ExperienceRecognising the common problems
Not panicking
Logical approachCheck configuration firstCheck locally first before blaming the peerTroubleshoot layer 1, then layer 2, then layer 3, etc
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 188
Troubleshooting Summary
Most troubleshooting is about:
Using the available toolsThe debugging tools on the router hardwareInternet Looking GlassesColleagues and their knowledgePublic mailing lists where appropriate
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 189
Closing Comments
Tutorial has covered the most common troubleshootingtechniques used by ISPs today
Once these have been mastered, more complex orarcane problems are easier to solve
Feedback and input for future improvements isencouraged and very welcome
© 2008 Cisco Systems, Inc. All rights reserved.APRICOT 2008 190
Troubleshooting BGP
The End!