66
CIS 188 CCNP TSHOOT (Troubleshooting) Ch. 6 Troubleshooting Addressing Services Rick Graziani Cabrillo College [email protected] Fall 2014
CIS 188 CCNP TSHOOT (Troubleshooting)Ch. 6 Troubleshooting Addressing
Services
Rick Graziani
Cabrillo College
Fall 2014
2
Materials
Book: Troubleshooting and Maintaining
Cisco IP Networks (TSHOOT) Foundation Learning Guide: Foundation learning for the CCNP TSHOOT 642-832
By Amir Ranjbar Book
ISBN-10: 1-58705-876-6 ISBN-13: 978-1-58705-876-9
eBook ISBN-10: 1-58714-170-1 ISBN-13: 978-1-58714-170-6
NAT/PAT Operation
Network Address Translation (NAT) was designed for IP version 4 (IPv4) Used for:
Address conservation Address hiding
NAT usually operates at the border of a network Translates the source address of the exiting IP packets that are private
addresses to public addresses before packets are forwarded.
3
NAT/PAT Operation Review – Cont.Order of operations with reference to NAT for incoming and outgoing traffic:
5
NAT Example
128.23.2.2 10.0.0.3 .... Data
DA SA
IP Header
128.23.2.2 179.9.8.80 .... Data
DA SA
IP Header
The translation from Private source IP address to Public source IP address.
1 2
1 2
6
NAT Example
Translation back, from Public destination IP address to Private destination IP address.
179.9.8.80 128.23.2.2 .... Data
DA SA
IP Header
10.0.0.3 128.23.2.2 .... Data
DA SA
IP Header
34
34
7
PAT Example
128.23.2.2 10.0.0.3 80 1331 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
128.23.2.2 10.0.0.2 80 1555 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
128.23.2.2 179.9.8.80 80 3333 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
128.23.2.2 179.9.8.80 80 2222 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
NAT/PAT table maintains translation of:
DA, SA, SP, Converted SP
1 2
Note: The router will try to maintain the originalTCP/UDP source port number if possible
8
PAT Example
179.9.8.80 128.23.2.2 3333 80 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
179.9.8.80 128.23.2.2 2222 80 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
10.0.0.3 128.23.2.2 1331 80 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
10.0.0.2 128.23.2.2 1555 80 Data
DA SA
IP Header
DP SP
TCP/UDPHeader
4 3
NAT/PAT table maintains translation of:
SA (DA), DA (SA), DP (SP)
Static NAT: Local and global addresses are mapped one to one.
Dynamic NAT: Local addresses are translated to a group or pool of global addresses. Still dealing with one-to-one translation once a global address has been
selected. NAT overloading:
Special type of dynamic NAT in which addresses are translated in a many-to-many fashion.
Also known as PAT, or Port Address Translation 9
Example: We want NAT to allow the first 31 hosts from each subnet on the inside to communicate with devices on the outside using a public IP address. Note: We will use the 172.16.10.0/24 network to simulate a public
address Range of public addresses: 172.16.10.1 through 172.16.10.63
10
Dynamic NATSA 172.16.10.1
interface ethernet 0
ip address 10.10.10.1 255.255.255.0
ip nat inside
interface ethernet 1
ip address 10.10.20.1 255.255.255.0
ip nat inside
interface serial 0
ip address 172.16.10.64 255.255.255.0
ip nat outside
ip nat inside source list 7 pool no-overload
!--- Indicates that any packets received on the inside interface that
!--- are permitted by access-list 7
!--- will have the source address translated to an address out of the
!--- NAT pool "no-overload".
access-list 7 permit 10.10.10.0 0.0.0.31
access-list 7 permit 10.10.20.0 0.0.0.31
!--- Access-list 7 permits packets with source addresses ranging from
!--- 10.10.10.0 through 10.10.10.31 and 10.10.20.0 through 10.10.20.31.
ip nat pool no-overload 172.16.10.1 172.16.10.63 prefix 24
!
!--- Defines a NAT pool named no-overload with a range of addresses
!--- 172.16.10.1 - 172.16.10.6311
Dynamic NAT (one-to-one) 172.16.10.1
172.16.10.2
172.16.10.3
172.16.10.4
interface ethernet 0
ip address 10.10.10.1 255.255.255.0
ip nat inside
interface ethernet 1
ip address 10.10.20.1 255.255.255.0
ip nat inside
interface serial 0
ip address 172.16.10.64 255.255.255.0
ip nat outside
ip nat inside source list 7 pool OVERLOAD overload
!
!--- Indicates that any packets received on the inside interface that
!--- are permitted by access-list 7 will have the source address
!--- translated to an address out of the NAT pool named ovrld.
!--- Translations will be overloaded which will allow multiple inside
!--- devices to be translated to the same valid IP address.
access-list 7 permit 10.10.10.0 0.0.0.31
access-list 7 permit 10.10.20.0 0.0.0.31
!--- Access-list 7 permits packets with source addresses ranging from
!--- 10.10.10.0 through 10.10.10.31 and 10.10.20.0 through 10.10.20.31
ip nat pool OVERLOAD 172.16.10.1 172.16.10.1 prefix 24
!
!--- Defines a NAT pool named ovrld with a range of a single IP
!--- address, 172.16.10.1.12
NAT Overloading (many-to-one)172.16.10.1
interface e 0
ip address 10.10.10.254 255.255.255.0
ip nat inside
interface s 0
ip address 172.16.130.2 255.255.255.0
ip nat outside
ip nat inside source list 7 pool test overload
ip nat inside source static 10.10.10.1 172.16.131.1
access-list 7 permit 10.10.10.0 0.0.0.255
ip nat pool test 172.16.131.2 172.16.131.2 netmask 255.255.255.013
172.16.131.1
172.16.131.2
Static and NAT Overloading (many-to-one)
Other examples http://www.cisco.com/en/US/tech/tk648/tk361/
technologies_tech_note09186a0080094e77.shtml#topic1
14
Advantages and disadvantages of implementing NAT
15
Some applications or protocols have direct conflict with Network Address Translation (NAT) or Port Address Translation (PAT).
Example: IPsec Virtual Private Networks (VPN) IPsec protocols encapsulate the original IP packet
The protocol type on the IP header changes (to ESP or AH) There is no TCP or UDP header next to the IP header.
Means that there is no port number for NAT/PAT to translate. Some mechanisms have been invented to allow IPsec and NAT to coexist. Those mechanisms include:
NAT Transparency or NAT traversal IPsec over TCP IPsec over UDP.
In certain cases you may still be required to disable NAT for VPN traffic, or create exceptions for it. 16
Troubleshooting Common NAT/PAT Issues
Some of the important NAT issues and considerations to keep in mind are: Having a diagram for the NAT configuration ACLs are used to tell the NAT device:
"what source IP addresses are to be translated” IP NAT pools are used to specify:
"to what those addresses translate” Marking the IP NAT inside interfaces and the IP NAT outside interfaces
correctly is very important NAT packets still have to obey routing protocols and reachability rules, so
make sure that every router knows how to reach the desired destinations. 17
Troubleshooting NAT problems: clear ip nat translations show ip nat translations show ip nat statistics debug ip nat debug ip packet [access-list] debug condition interface interface show debug condition
18
Troubleshooting Example: NAT/PAT Problems
19
Router R1 can ping R4, but router R1 cannot ping R3. There are no routing protocols running in any of the routers. R1 uses R2 as its gateway of last resort. The objective is to restore end-to-end connectivity from R1 to all
destinations.
20
21
Looks good
Looks goodpool
• This command also shows the full NAT configuration.
• In this instance, we have both dynamic and static NAT configured.
• The only indication that there are static translations configured is the reference to “1 static” in the “Total active translations” line.
• In order to see the static translation, the show ip nat translations command is used.
The only entry in the NAT translation table is the static translation for R1's Fa0/0 interface 10.10.10.1 into 172.16.6.1.
This translation ‘may’ be causing the problem. Typical issues with static translations occur when:
there is no route back to the statically translated address, or when the statically selected global address overlaps with an available
address in the dynamic address pool. 22
10.10.10.1 172.16.6.1
The next step, verify whether packets leaving R1 actually reach R3 or not. This can help us discover if the problem is with NAT a routing problem. Use icmp debugging on R3 with the debug ip icmp command. Next, we ping R3 from R1 and observe the out of the debug on R3. Based on the output of the debug ip icmp command shown, the icmp echo
requests reach R3, but R3’s icmp echo replies do not reach R1. The NAT translation is working but there is a routing issue on R3 toward the
172.16.6.0 destination. 23
?R3# debug ip icmpICMP packet debugging is on
R1# ping 172.16.11.3Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 172.16.11.3, timeout is 2 seconds:.....Success rate is 0 percent (0/5)
R3#*Aug 23 13:54:00.556: ICMP: echo reply sent, src 172.16.11.3, dst 172.16.6.1*Aug 23 13:54:02.552: ICMP: echo reply sent, src 172.16.11.3, dst 172.16.6.1*Aug 23 13:54:04.552: ICMP: echo reply sent, src 172.16.11.3, dst 172.16.6.1*Aug 23 13:54:06.552: ICMP: echo reply sent, src 172.16.11.3, dst 172.16.6.1*Aug 23 13:54:07.552: ICMP: echo reply sent, src 172.16.11.3, dst 172.16.6.1
This is verified using the show ip route command on R3. Because there are no routing protocols in use, we fix the routing problem by
entering a static route in R3’s routing table. Now we can check to see if the connectivity problem between R1 and R3
has been corrected. The problem was not directly a NAT problem; it was a routing issue.
24
?
R3# show ip route 172.16.6.0 255.255.255.0% Subnet not in table
R3# configure terminalR3(config)# ip route 172.16.6.0 255.255.255.0 172.16.11.2R3(config)# exit
R1# ping 172.16.11.3Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 172.16.11.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 msR1#
When NAT is deployed, a set of inside local addresses are normally translated to a set of inside global addresses.
The inside global addresses may not be assigned to any physical networks, yet they need to be advertised so that outside devices know about those addresses and know how to forward packets to those networks.
25
Troubleshooting Example: NAT Problem Caused by an Inaccurate Access List
26
We are told that administrators are unable to use Secure Shell (SSH) from the 10.10.10.0/24 network to routers R3 or R4
But they can accomplish connectivity (ping, etc) from the R1 loopbacks. The routing protocol used is single area OSPF Our mission is to restore end-to-end connectivity and make sure SSH is
operational to support management processes.
27
SSH
SSH
ICMP
ICMP
Is SSH traffic is being filtered somewhere? Or is this simply be a routing problem - traffic is either not reaching the
destination, or it is not coming back? Not a routing problem
28
Our follow-the-path strategy points to looking at the intermediate routers all the way up to the destination (R3).
We will use a clever tool to discover the potential filter: ip tcp debug transactions
Try the SSH connection again and observe the output of debug. The results indicate that the attempt made by R1 to setup a TCP session
with R3 failed because the remote device (R3) responded with a TCP RST (reset).
29
Focus on R3: the output of show ip int serial 0/1/0 shows that an access-list called FIREWALL-INBOUND is applied to serial 0/1/0 interface on the inbound direction.
show access-lists command and the access list looks correct: statement number 30 permits TCP connection to 172.16.11.3 TCP port number 22 (SSH). 30
Cautiously make use of debug ip packet on R3. Re-attempt the SSH session from R1 to R3 and observe the debug output on
R3. The SSH attempt from R1 fails again, but:
Security message (%SEC-6-IPACCESSLOGP) on R3 states that the denied TCP with:
Source IP address 10.10.10.1 and source port number 29832 Destination IP address 172.16.11.3 and destination port number 2222
Now we know why the packet is denied, destination port is 2222 instead of 22. 31
We have to find out which device has translated the port number from 22 to 2222.
Prime suspect is NAT on R2. Cautiously, we use debug ip nat on R2 Re-attempt SSH from R1 to R3. To confirm our findings, we also enter the show ip nat translations
command on R2. 32
We are using TCP 2222 (destination port not the NAT source port); but the access-list on R3 is permitting TCP 22.
The problem was not the NAT configuration, but a lack of synchronization between the configuration teams:
The configuration on R2 is doing port mapping to a custom port (2222), but the access-list configuration on R3 did not consider or account for the custom port.
We need to correct the FIREWALL-INBOUND on R3 33
R2# debug ip natIP NAT debugging is onR2#*Aug 23 16:28:31.731: NAT*: TCP s=555 55587, d=22->2222
R3# conf tEnter configuration commands, one per line. End with CNTL/Z.R3(config)# ip access-list exten FIREWALL-INBOUNDR3(config-ext-nacl)# permit tcp any host 172.16.11.3 eq 22R3(config-ext-nacl)# endR3#
The next step is to correct the FIREWALL-INBOUND on R3 and re-attempt SSH from R1 to R3.
34
R3# conf tEnter configuration commands, one per line. End with CNTL/Z.R3(config)# ip access-list exten FIREWALL-INBOUNDR3(config-ext-nacl)# permit tcp any host 172.16.11.3 eq 2222R3(config-ext-nacl)# endR3#
SSH attempt is successful now. - The problem was not the NAT configuration. The configuration on R2 was doing port mapping to a custom port (2222) but
the access-list configuration on R3 did not consider or account for the custom port. 35
R3# conf tEnter configuration commands, one per line. End with CNTL/Z.R3(config)# ip access-list exten FIREWALL-INBOUNDR3(config-ext-nacl)# permit tcp any host 172.16.11.3 eq 2222R3(config-ext-nacl)# endR3#
R1# ssh -l user 172.16.11.3Password:*Aug 23 16:30:42.604: TCP: Random local port generated 43884, network 1*Aug 23 16:30:26.604: TCB63BF854C created*Aug 23 16:30:26.604: TCB63BF854C bound to UNKNOWN.43884*Aug 23 16:30:26.604: TCB63BF854C setting property TCP_TOS (11) 62AF6D55*Aug 23 16:30:26.604: Reserved port 43884 in Transport Port Agent for TCP IP type 1*Aug 23 16:30:26.604: TCP: sending SYN, seq 1505095793, ack 0*Aug 23 16:30:26.604: TCP0: Connection to 172.16.11.3:22, advertising MSS 536*Aug 23 16:30:26.608: TCP0: state was CLOSED -> SYNSENT [43884 ->172.16.11.3(22)]*Aug 23 16:30:26.608: TCP0: state was SYNSENT -> ESTAB [43884 ->172.16.11.3(22)]*Aug 23 16:30:26.608: TCP: tcb 63BF854C connection to 172.16.11.3:22, peer MSS536, MSS is 536*Aug 23 16:30:26.608: TCB63BF854C connected to 172.16.11.3.22
Reviewing DHCP Operation
Reviewing DHCP Operation
37
These commands are most helpful to know during the troubleshooting process, for example.
DHCPDECLINE – client-to-server message - The client through some other means has discovered that the IP address is already in use.
DHCPNAK - DHCP server refuses the request for a certain configuration parameter.
Three roles router takes in DHCP
38
The most common reasons for problems are configuration issues. This can result in a multitude of symptoms, such as:
Clients not obtaining IP information from the server Client requests not reaching the server across a DHCP relay agent Clients failing to obtain DHCP options and extensions.
Another issue related to DHCP Relay Agent is that enabling a router interface with the ip helper-address interface command
Allows you to control which broadcast packets and which protocols are forwarded.
Address: Destination broadcast or host address to be used when forwarding UDP broadcasts.
Forwards six protocols (not just DHCP) TFTP (port 69) DNS (port 53) Time Service (port 37) NetBIOS Name Service and Datagram Service (ports 137 and 138) TACACS (port 49) DHCP/BOOTP Client and Server (ports 67 and 68)
If other protocols do not require this service, forwarding their requests should be disabled using the no ip forward-protocol udp port-number global configuration mode command.
39
Router(config-if)# ip helper-address address
DHCP options deliver parameters in addition to the traditional IP address, subnet mask, default-gateway, and DNS server address.
82 - Used for 911 purposes, in order to know the switch port that an IP phone connects to – the 911 caller’s exact location.
40
41
show ip dhcp server statistics: Displays counts for server statistics and messages sent and received for an IOS-based DHCP server.
show ip dhcp binding: Displays DHCP binding information for IP address assignment and subnet allocation.
show ip dhcp conflict: Displays address conflicts found by a Cisco IOS DHCP server when addresses are offered to the client.
show ip dhcp pool name: Displays the subnets allocated and the current utilization level for the pool or all the pools if the name argument is not used.
show ip dhcp database: Displays server database agent information: URL: Specifies the remote file used to store automatic DHCP bindings Read/written: The last date and time bindings were read/written from the file server Status: Indication of whether the last read or write of host bindings was
successful Delay: The amount of time (in seconds) to wait before updating the database Timeout: The amount of time (in seconds) before the file transfer is aborted Failures/Successes: The number of failed/successful file transfers
DHCP Troubleshooting Commands
Three DHCP address pools are created: network 172.16.0.0, subnet 172.16.1.0 and subnet 172.16.2.0
Attributes from network 172.16.0.0 - such as the domain name, DNS server, NetBIOS name server, and NetBIOS node type are inherited in subnetworks 172.16.1.0 and 172.16.2.0.
In each pool, clients are granted 30-day leases and all addresses in each subnetwork, except the excluded addresses, are available to the DHCP Server for assigning to clients.
Excluded addresses include default-gateway, DNS and NETBIOS servers 42
ip dhcp excluded-address 172.16.1.100 172.16.1.103
ip dhcp excluded-address 172.16.2.100 172.16.2.103
!
ip dhcp pool 0
network 172.16.0.0 /16
domain-name cisco.com
dns-server 172.16.1.102 172.16.2.102
netbios-name-server 172.16.1.103 172.16.2.103
netbios-node-type h-node
!
ip dhcp pool 1
network 172.16.1.0 /24
default-router 172.16.1.100 172.16.1.101
lease 30
!
ip dhcp pool 2
network 172.16.2.0 /24
default-router 172.16.2.100 172.16.2.101
lease 30
DHCP Troubleshooting Issues
Troubleshooting can be related to DHCP security efforts. Automatic addressing is accomplished through DHCP. Security is accomplished through DHCP snooping. Some specific issues related to DHCP snooping:
Improper configuration of the DHCP snooping trust boundaries Failure to configure DHCP snooping on certain VLANs Improper configuration of the DHCP snooping rate limits Performance degradation
Poor planning of DHCP snooping can result in DHCP transactions being blocked or affected.
DHCP Troubleshooting Issues – Cont.
DHCP troubleshooting questions to ask:Where are the DHCP servers and clients located?Are DHCP relay agents configured?What are the DHCP pool sizes? Are they sufficient?Are there any DHCP option compatibility issues?Are there any ACLs or firewalls filtering UDP port 67 or UDP port 68?Are there any active DHCP DoS attacks?Is forwarding disabled on the router acting as DHCP Relay Agent for any UDP ports (using the Cisco IOS no ip forward-protocol udp port command)?Is the ip helper-address command applied to correct router interfaces?Is DHCP snooping configured?
Troubleshooting Example: DHCP Problems
45
R1 provides DHCP services to the clients in the 10.1.1.0 subnet. (Not R4) The DHCP clients in this example are routers R2 and R3. It is reported that R1 is no longer providing reliable DHCP services:
The clients are unable to renew their IP addresses.
46
Check R2 and R3 to make sure that they are configured as a DHCP clients. The output of the show ip interfaces brief command shows that interface
fa0/0 is configured as a DHCP client and it shows an “unassigned” IP address.
Because multiple clients are having the same problem, it is reasonable to suspect the problem originates elsewhere. 47
R2# show ip int briefInterface IP-Address OK? Method Status ProtocolFastEthernet0/0 unassigned YES DHCP up upFastEthernet0/1 unassigned YES NVRAM administratively down downSerial0/0/0 unassigned YES NVRAM administratively down downSerial0/0/1 unassigned YES NVRAM administratively down down
R3# show ip int briefInterface IP-Address OK? Method Status ProtocolFastEthernet0/0 unassigned YES DHCP up upFastEthernet0/1 unassigned YES NVRAM administratively down downSerial0/0/0 unassigned YES NVRAM administratively down downSerial0/0/1 unassigned YES NVRAM administratively down down
DHCP Troubleshooting Example 1 – Cont.
R3# debug dhcp detailDHCP client activity debugging is on (detailed)R3#
*Aug 23 17:32:37.107: Retry count: 1 Client-ID: cisco-0019.5592.a442-Fa0/0*Aug 23 17:32:37.107: Client-ID hex dump: 636973636F2D303031392E353539322E*Aug 23 17:32:37.107: 613434322D4551302F30*Aug 23 17:32:37.107: Hostname: R3*Aug 23 17:32:37.107: DHCP: SDiscover: sending 291 byte length DHCP packet*Aug 23 17:32:37.107: DHCP: SDiscover 291 bytes*Aug 23 17:32:37.107: B cast on FastEthernet0/0 interface from 0.0.0.0*Aug 23 17:32:40.395: DHCP: SDiscover attempt #2 for entry:*Aug 23 17:32:40.395: Temp IP addr: 0.0.0.0 for peer on Interface: FastEthernet0/0*Aug 23 17:32:40.395: Temp sub net mask: 0.0.0.0*Aug 23 17:32:40.395: DHCP Lease server: 0.0.0.0, state: 1 Selecting*Aug 23 17:32:40.395: DHCP transaction id: 13BA*Aug 23 17:32:40.395: Lease: 0 secs, Renewal: 0 secs, Rebind: 0 secs*Aug 23 17:32:40.395: Next timer fires after: 00:00:04*Aug 23 17:32:40.395: Retry count: 2 Client-ID: cisco-0019.5592.a442-Fa0/0*Aug 23 17:32:40.395: Client-ID hex dump: 636973636F2D303031392E353539322E*Aug 23 17:32:40.395: 613434322D4551302F30<output omitted>*Aug 23 17:32:44.395: Hostname: R3*Aug 23 17:32:44.395: DHCP: SDiscover: sending 291 byte length DHCP packet*Aug 23 17:32:44.395: DHCP: SDiscover 291 bytes*Aug 23 17:32:44.395: B cast on FastEthernet0/0 interface from 0.0.0.0*Aug 23 17:32:48.395: DHCP: Qscan: Timed out Selecting state%Unknown DHCP problem... No allocation possible*Aug 23 17:32:57.587: DHCP: waiting for 60 seconds on interface FastEthernet0/0
debug dhcp detail command on R3: Finds that the DHCP DISCOVER messages are generated out of interface
fa0/0, but no DHCP offers are received back from the DHCP server. “Timed out Selecting state” message, followed by “No allocation
possible.”
DHCP Troubleshooting Example 1 – Cont.
R1# show ip int briefInterface IP-Address OK? Method Status ProtocolFastEthernet0/0 10.1.1.1 YES manual up upFastEthernet0/1 unassigned YES NVRAM administratively down downSerial0/0/0 unassigned YES NVRAM administratively down downSerial0/0/1 unassigned YES NVRAM administratively down down
Next, check router R1, the DHCP server. Apply a bottom-up approach and use the show ip int brief command to verify
that Layers 1 and 2 are operational on the documented DHCP server. R1's Fa0/0 interface is UP, with the IP address 10.1.1.1/24.
DHCP Troubleshooting Example 1 – Cont.R1# show ip dhcp server statisticsMemory usage 9106Address pools 1Database agents 0Automatic bindings 0Manual bindings 0Expired bindings 0Malformed messages 0Secure arp entries 0
Message ReceivedBOOTREQUEST 0DHCPDISCOVER 1DHCPREQUEST 1DHCPDECLINE 0DHCPRELEASE 0DHCPINFORM 0Message SentBOOTREPLY 0DHCPOFFER 1DHCPACK 1DHCPNAK 0
shows that one address pool configured as part of the service and very few DHCP messages sent and received.
So it looks to be in good shape, just not very active.
DHCP Troubleshooting Example 1 – Cont.
R1# sh ip dhcp pool
Pool vlan10 :
Utilization mark (high/low) : 100/0
Subnet size (first/next) : 0/0
Total addresses : 254
Leased addresses : 0
Pending event : none
1 subnet is currently in the pool :
Current index IP address range Leased addresses
10.1.1.12 10.1.1.1 -10.1.1.254 0
Verify the address pool (scope) configuration to determine if it is either misconfigured or exhausted, using the show ip dhcp pool command.
The results show one address pool, vlan10. The number of leased addresses is 0, meaning that there are no leased
addresses. There 254 addresses in this pool, but none are allocated at the moment. Both the DHCP server and the DHCP clients have the correct configurations, and
are operationally UDP at physical and data-link layers.
DHCP Troubleshooting Example 1 – Cont.
R1# show ip socketsProto Remote Port Local Port In Out Stat TTY OutputIF88 --listen-- 10.1.1.1 10 0 0 0 017 --listen-- 10.1.1.1 161 0 0 1001 017 --listen-- 10.1.1.1 162 0 0 1011 017 --listen-- 10.1.1.1 57767 0 0 1011 017 --listen-- --any-- 161 0 0 20001 017 --listen-- --any-- 162 0 0 20011 017 --listen-- --any-- 60739 0 0 20011 0
We use the show ip socket command to see the active ports on R1, the DHCP server.
The show ip socket command is not frequently used by network administrators, but it is very handy in monitoring the open ports on a router
There is no entry for UDP port 67 (DHCP Server). This is certainly a problem.
DHCP Troubleshooting Example 1 – Cont.R1# conf t
R1(config)# service dhcp
R1# show ip sockets
Proto Remote Port Local Port In Out Stat TTY OutputIF
88 --listen-- 10.1.1.1 10 0 0 0 0
17 --listen-- 10.1.1.1 161 0 0 1001 0
17 --listen-- 10.1.1.1 162 0 0 1011 0
17 --listen-- 10.1.1.1 57767 0 0 1011 0
17 --listen-- --any-- 161 0 0 20001 0
17 --listen-- --any-- 162 0 0 20011 0
17 --listen-- --any-- 60739 0 0 20011 0
17 0.0.0.0 0 10.1.1.1 67 0 0 2211 0
We enable the DHCP service using the service dhcp command. After enabling this service you must wait a few seconds, because DHCP clients
retry at different intervals. Use the show ip socket command again and now see port 67 as an active port on
R1. Finally, we check routers R2 and R3 (the DHCP clients) and see that they
successfully obtained IP addresses.
DHCP Troubleshooting Example: Duplicate Client IP Addresses
54
The IP address of router R1 on the fast Ethernet interface was changed from 10.1.1.100 to 10.1.1.1 to comply with the new addressing scheme and policies of the network.
This policy states that all branch routers will have the first IP address on any subnet that is being assigned to a network segment.
After the change, some DHCP clients are reporting duplicated IP addresses. Clients state that this happens sporadically, a few times a week.
55
DHCP Troubleshooting Example 2 – Cont.
R1# show running-config | beg ip dhcp pool
ip dhcp pool vlan10network 10.1.1.0 255.255.255.0default-router 10.1.1.1lease 3
One piece of information we have is that the IP address duplication happens sporadically, and one host at a time.
Knowing that, perhaps our first order of business is to look at the lease times, and see if they match the frequency of the symptoms.
The Cisco IOS DHCP default values is 1 day. The output of this command shows the vlan10 DHCP pool with a lease time
of 3 days. The DHCP pool seems to be correct.
DHCP Troubleshooting Example 2 – Cont.
R1# show ip dhcp conflictIP address Detection method Detection time VRF10.1.1.1 Gratuitous ARP Aug 23 2009 06:28 PM10.1.1.3 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.3 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.4 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.5 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.6 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.7 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.8 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.9 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.10 Gratuitous ARP Aug 23 2009 06:29 PM10.1.1.11 Gratuitous ARP Aug 23 2009 06:29 PM--More--
show ip dhcp conflict command will tell us whether the DHCP server has found overlap or duplication in the IP addresses that it has assigned.
One of the many conflicting addresses is 10.1.1.1, which is the new ip address of router R1 (the dhcp server itself) on interface fa0/0.
However, we know that the DHCP server should not provide its own IP address to its clients!
Many devices such as servers and printers are usually configured as DHCP clients and have static IP addresses.
If their addresses are not excluded from the DHCP dynamic pool, there will definitely be conflict problems.
We must check and verify which IP addresses are being excluded on R1, the DHCP server.
We do that using the show running | include excluded command. The only IP address excluded from the DHCP dynamic pool is 10.1.1.100,
which is R1’s old address. 58
R1# sh run | inc excluded
ip dhcp excluded-address 10.1.1.100
R1#
We need to exclude the range of addresses that are meant to be statically assigned.
This range is 10.1.1.1 to 10.1.1.20 To ensure that the users will receive unique addresses from the DHCP
server and will not incur any more address conflicts, we must renew IP address leases on all DHCP clients, especially those that have experienced conflicts before. (Success!)
59
R1# conf tEnter configuration commands, one per line. End with CNTL/Z.
R1(config)# no ip dhcp excluded-address 10.1.1.100R1(config)# ip dhcp excluded-address 10.1.1.1 10.1.1.20R1(config)# endR1#
DHCP Troubleshooting Example 3: Relay Agent Issue
R4 is a centrally located DHCP server. The DHCP clients in network segment 10.1.1.0 are unable to obtain IP
address and other parameters. R2 is a DHCP client that is having trouble acquiring ip address. R1 is supposed to act as a relay agent and forward DHCP messages
between local clients and the DHCP server (R4).
There are several possible causes: The clients could be misconfigured or faulty The relay agent could be not configured or misconfigured The server could be misconfigured or exhausted (or faulty/disabled) There could be network problems or filtering/security barriers.
Because multiple clients are having the same problem, it is possible that they are all misconfigured, but that is not likely.
It is simpler to check the relay agent. If we find no problem with the relay agent, we can then check the DHCP
server. 61
?
?
DHCP Troubleshooting Example 3 – Cont.
R1# debug ip udpUDP packet debugging is onR1#R1#*Aug 23 19:01:05.303: UDP: rcvd src-0.0.0.0(68), dst=255.255.255.255(67),length=584*Aug 23 19:01:05.303: UDP: broadcast packet dropped, src=0.0.0.0,dst=192.168.1.255*Aug 23 19:01:08.911: UDP: rcvd src-0.0.0.0(68), dst=255.255.255.255(67),length=584*Aug 23 19:01:08.911: UDP: broadcast packet dropped, src=0.0.0.0,dst=192.168.1.255*Aug 23 19:01:12.911: UDP: rcvd src-0.0.0.0(68), dst=255.255.255.255(67),length=584*Aug 23 19:01:12.911: UDP: broadcast packet dropped, src=0.0.0.0,dst=192.168.1.255<output omitted>
One of the quickest ways to verify DHCP relay agent operations is using the debug ip udp command.
R1 is certainly receiving DHCP requests but is dropping them. The UDP/IP packets shown have a source address of 0.0.0.0, destination address
of 255.255.255.255 with source UDP port of 68 (DHCP client) and destination UDP port of 67 (DHCP server).
The problem could be that the fa 0/0 interface facing the DHCP client is missing the ip helper-address command pointing to 192.168.1.4.
DHCP Troubleshooting Example 3 – Cont.
R1(config)# int fa0/0R1(config-if)# ip helper-address 192.168.1.4R1(config-if)# end
Checking the configuration reveals that this command is indeed missing, so adding this command is the first thing we have to do.
debug ip udp command on R4, shows the DHCP Requests are being forwarded to R1
DHCP messages fro the server are being received Finally, we verify the status of the DHCP clients, such as R2, in the 10.1.1.0 subnet,
and see that they are acquiring IP address and other parameters from the DHCP server.
R4# debug ip udpUDP packet debugging is onR4#*Aug 23 19:31:39.303: UDP: sent src=0.0.0.0(67), dst=255.255.255.255(68),length=308*Aug 23 19:31:39.303: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67),length=584*Aug 23 19:31:39.303: UDP: sent src=0.0.0.0(67), dst=255.255.255.255(68),length=308*Aug 23 19:31:40.159: UDP: rcvd src=0.0.0.0(68), dst=192.168.1.4(67), length=584*Aug 23 19:31:44.159: UDP: rcvd src=0.0.0.0(68), dst=192.168.1.4(67), length=584*Aug 23 19:31:46.307: UDP: rcvd src=10.1.1.11(53470), dst=255.255.255.255(69),length=30*Aug 23 19:31:49.307: UDP: rcvd src=10.1.1.11(53470), dst=255.255.255.255(69),length=30<output omitted>*Aug 23 19:32:28.439: UDP: rcvd src=10.1.1.11(53470), dst=255.255.255.255(69),length=29*Aug 23 19:32:31.439: UDP: rcvd src=10.1.1.11(53470), dst=255.255.255.255(69),length=29*Aug 23 19:32:35.439: UDP: rcvd src=10.1.1.11(53470), dst=255.255.255.255(69),length=29*Aug 23 19:32:37.011: UDP: rcvd src=0.0.0.0(68), dst=192.168.1.4(67), length=584
65
IPv6
Next: Series of IPv6 videos from the Academy Conference I presented at in 2012.
See my web page or summary email for links.
CIS 188 CCNP TSHOOT (Troubleshooting)Ch. 6 Troubleshooting Addressing
Services
Rick Graziani
Cabrillo College
