Network TroubleshootingNetwork Troubleshooting

Chapter 15

ObjectivesObjectives• Describe appropriate troubleshooting

tools and their functions• Analyze and discuss the troubleshooting

process• Tackle a variety of troubleshooting

scenarios

OverviewOverview

Introduction to Network Introduction to Network TroubleshootingTroubleshooting

Three Parts to Chapter 15Three Parts to Chapter 15

• Troubleshooting Tools• The Troubleshooting Process• Troubleshooting Scenarios

Troubleshooting ToolsTroubleshooting Tools

• Hardware Tools– Cable Tester, TDR, and OTDR– Certifiers– Voltage Event Recorder/Temperature Monitor– Protocol Analyzer– Cable Stripper/Snips– Multimeter– Tone Probe and Tone Generator– Butt Set– Punchdown Tool

• Hardware Tools– Cable Testers, TDRs, and OTDRs

• Devices to test for broken cables–Cable testers

» Continuity problems and wire map

–TDRs time domain reflectometer» Locates copper cable breaks

–OTDRs» Locates fiber-optic cable breaks

Figure 15.1 Typical cable tester

Figure 15.2 An EXFO AXS-100 OTDR (photo courtesy of EXFO)

• Certifiers– Only use for slowdowns, not disconnects– Require a loopback on the far end– Ensure a cable can handle its rated capacity– Problems that reduce cable capacity

• Cross talk• Attenuation• Interference

• Voltage Event Recorder/Temperature Monitor– Detect power and heat problems– Cause intermittent problems– Heat problems in server rooms – Monitor power with voltage event recorder– Monitor temperature with temperature

monitor

• Protocol Analyzers– Monitor protocols at different layers

• Application, Session, Network, Data Link– Both hardware and software tools

• When to Use a Protocol Analyzer– You need to see data to analyze the problem

• A session fails to start• A DNS server fails to respond• Confusing information appears on the network• You suspect a rogue DHCP server exists• Excess traffic is slowing down the network

• Cable Stripper or Snip– Enable you to make UTP cables– Also need crimpers– Often combined in one tool

Figure 15.3 A cable stripping and crimping tool

• Multimeters– Test AC and DC voltage– Test resistance– Test Continuity– A great fallback when you do not have

a cable tester

• Tone Probes and Tone Generators– Work together to identify opposite ends of

unlabeled cable runs– Tone generator puts a signal (tone) on wire– Tone probe on opposite end detects the

signal

• Butt Sets– Hand set– Use to tap into a 66- or 100-block to

test a line

• Punchdown Tools– Put UTP wires into 66- and 100-blocks– Repunch a connection to make sure

contacts are set

Figure 15.4 A punchdown tool in action

• Software Tools– TRACERT/TRACEROUTE– IPCONFIG/IFCONFIG– PING and ARP PING– NSLOOKUP/DIG– HOSTNAME– MTR– ROUTE– NBTSTAT– NETSTAT– Packet Sniffer– Port Scanners

• TRACERT/TRACEROUTE– Traces all routers between two points– Use to discover where a problem lies– Problem is just beyond the last router

displayed before error– Some routers block TRACEROUTE packets– If TRACEROUTE is blocked, try PATHPING

Tracing route to adsl-208-190-121-38.dsl.hstntx.swbell.net[208.190.121.38] over a maximum of 30 hops:1 1 ms <1 ms 1 ms Router.totalhome[192.168.4.1]2 38 ms 41 ms 70 ms adsl-208-190-121-

38.dsl.hstntx.swbell.net [208.190.121.38]

Listing 15-1 Sample TRACEROUTE output

• IPCONFIG/IFCONFIG– Displays IP settings– IPCONFIG without parameters

• Basic information only– Ipconfig /all gives configuration details

Ethernet adapter Main:

Connection-specific DNS Suffix . :IPv6 Address . . . . . . . . . . : 2001:470:bf88:1:fc2d:aeb2:99d2:e2b4Temporary IPv6 Address . . . . . : 2001:470:bf88:1:5e4:c1ef:7b30:ddd6Link-local IPv6 Address. . . . . : fe80::fc2d:aeb2:99d2:e2b4%8IPv4 Address . . . . . . . . . . : 192.168.4.27Subnet Mask . . . . . . . . . . : 255.255.255.0Default Gateway . . . . . . . . : fe80::223:4ff:fe8c:b720%8

192.168.4.1Tunnel adapter Local Area Connection* 6:

Media State . . . . . . . . . . : Media disconnectedConnection-specific DNS Suffix . :

Listing 15-2 Sample IPCONFIG output

eth0 Link encap:Ethernet HWaddr 00:02:b3:8a:7d:aeinet addr:192.168.4.19 Bcast:192.168.4.255 Mask:255.255.255.0inet6 addr: 2001:470:bf88:1:202:b3ff:fe8a:7dae/64 Scope:Globalinet6 addr: fe80::202:b3ff:fe8a:7dae/64 Scope:LinkUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1RX packets:2206320 errors:0 dropped:0 overruns:0 frame:0TX packets:925034 errors:0 dropped:0 overruns:0 carriers:0collisions:0 txqueuelen:1000RX bytes:292522698 (292.5 MB) TX bytes:132985596 (132.9 MB)

lo Link encap:Local Loopbackinet addr:127.0.0.1 Mask:255.0.0.0inet6 addr: ::1/128 Scope:HostUP LOOPBACK RUNNING MTU:16436 Metric:1RX packets:15414 errors:0 dropped:0 overruns:0 frame:0TX packets:15414 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:0RX bytes:1006671 (1.0 MB) TX bytes:1006671 (1.0 MB)

Listing 15-3 Sample IFCONFIG output

• PING – Queries by name or IP address– Uses ICMP packets– Works across routers– Problem: devices can block ICMP

Pinging 192.168.4.19 with 32 bytes of data:Reply from 192.168.4.19: bytes=32 time<1ms TTL=64Reply from 192.168.4.19: bytes=32 time<1ms TTL=64Reply from 192.168.4.19: bytes=32 time<1ms TTL=64Reply from 192.168.4.19: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.4.19: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms

Listing 15-4 Sample PING output

• ARP PING – Queries by IP address– Uses ARP packets– Problem: does not cross routers– Only on UNIX and UNIX-like systems

ARPING 192.168.4.27 from 192.168.4.19 eth0

Unicast reply from 192.168.4.27 [00:1D:60:DD:92:C6] 0.875ms

Unicast reply from 192.168.4.27 [00:1D:60:DD:92:C6] 0.897ms

Unicast reply from 192.168.4.27 [00:1D:60:DD:92:C6] 0.924ms

Unicast reply from 192.168.4.27 [00:1D:60:DD:92:C6] 0.977ms

Listing 15-5 Sample ARPING output

• NSLOOKUP/DIG– Both diagnose DNS problems– NSLOOKUP (all operating systems)

• Poor tool – considered obsolete• Without switches, provides name and

IP address of default DNS server– DIG – more powerful

• Everyone but Windows

dig mx totalsem.com

; <<>> DIG 9.5.0-P2 <<>> mx totalsem.com

;; global options: printcmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6070

;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:

;totalsem.com. IN MX

;; ANSWER SECTION:

totalsem.com. 86400 IN MX 10

mx1c1.megamailservers.com.

totalsem.com. 86400 IN MX 100

mx2c1.megamailservers.com.

totalsem.com. 86400 IN MX 110

mx3c1.megamailservers.com.

Listing 15-6 The DIG command

Listing 15-7 Output for the DIG command

• HOSTNAME– Simplest of all utilities– Returns name of host from which it runs– HOSTNAME sample output

c:\>

c:\>hostname

Mike-win7beta

• My Traceroute (MTR)– Dynamic (keeps running)– Equivalent to TRACEROUTE– Not available in Windows

My traceroute [v0.73]totaltest (0.0.0.0)Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev1. Router.totalhome 0.0% 5 0.8 0.8 0.7 0.9 0.12. adsl-208-190-121-38.dsl.hstntx.s 0.0% 4 85.7 90.7 69.5 119.2 20.8

Listing 15-9 Sample MTR output

• ROUTE– Display and edit local routing table– Type route print

===========================================================================Interface List 8 ...00 1d 60 dd 92 c6 ...... Marvell 88E8056 PCI-E Ethernet Controller 1 ........................... Software Loopback Interface 1===========================================================================IPv4 Route Table===========================================================================Active Routes:Network Destination Netmask Gateway Interface Metric 0.0.0.0 0.0.0.0 192.168.4.1 192.168.4.27 10 127.0.0.0 255.0.0.0 On-link 127.0.0.1 306 127.0.0.1 255.255.255.255 On-link 127.0.0.1 306 127.255.255.255 255.255.255.255 On-link 127.0.0.1 306 169.254.0.0 255.255.0.0 On-link 192.168.4.27 286 169.254.214.185 255.255.255.255 On-link 169.254.214.185 276 169.254.255.255 255.255.255.255 On-link 192.168.4.27 266 192.168.4.0 255.255.255.0 On-link 192.168.4.27 266 192.168.4.27 255.255.255.255 On-link 192.168.4.27 266 192.168.4.255 255.255.255.255 On-link 192.168.4.27 266 224.0.0.0 240.0.0.0 On-link 127.0.0.1 306 224.0.0.0 240.0.0.0 On-link 169.254.214.185 276 224.0.0.0 240.0.0.0 On-link 192.168.4.27 266 255.255.255.255 255.255.255.255 On-link 127.0.0.1 306 255.255.255.255 255.255.255.255 On-link 169.254.214.185 276 255.255.255.255 255.255.255.255 On-link 192.168.4.27 266===========================================================================Persistent Routes:None

Listing 15-10 Sample ROUTE PRINT output

• NBTSTAT– Windows only– Command-line equivalent of My Network

Places– Must use a switch–nbtstat –n shows local NetBIOS names

Main:

Node IpAddress: [192.168.4.27] Scope Id: []

NetBIOS Local Name Table

Name Type Status

---------------------------------------------

MIKESPC <00> UNIQUE Registered

TOTALHOME <00> GROUP Registered

MIKESPC <20> UNIQUE Registered

TOTALHOME <1E> GROUP Registered

Listing 15-11 Sample NBTSTAT output

• NETSTAT– Shows current state of running IP processes– Shows what sessions are active– Provides statistics based on ports or

protocols– Type netstat to show only current sessions– Type netstat –r to show routing table

(like route print)

Active ConnectionsProto Local Address Foreign Address StateTCP 127.0.0.1:27015 MikesPC:51090 ESTABLISHEDTCP 127.0.0.1:51090 MikesPC:27015 ESTABLISHEDTCP 127.0.0.1:52500 MikesPC:52501 ESTABLISHEDTCP 192.168.4.27:54731 72-165-61-141:27039 CLOSE_WAITTCP 192.168.4.27:55080 63-246-140-18:http CLOSE_WAITTCP 192.168.4.27:56126 acd4129913:https ESTABLISHEDTCP 192.168.4.27:62727 TOTALTEST:ssh ESTABLISHEDTCP 192.168.4.27:63325 65.54.165.136:https TIME_WAITTCP 192.168.4.27:63968 209.8.115.129:http ESTABLISHED

Listing 15-12 Sample NETSTAT output

• Packet Sniffer– A.k.a. protocol analyzer or packet analyzer– Intercepts and logs network packets– Many choices– Software example: Wireshark– Dedicated hardware devices

Figure 15.5 Wireshark in action

• Port Scanners– Probes remotes system’s ports– Logs state of scanned ports– Good use: find unintentionally opened

ports in order to close– Bad use: find open ports and use to break in– NMAP runs on UNIX– Angry IP Scanner for Windows

Figure 15.6 Angry IP Scanner

The Troubleshooting The Troubleshooting ProcessProcess

Memorize these problem analysis steps:1. Information gathering—identify symptoms and problems2. Identify the affected areas of the network3. Establish if anything has changed4. Establish the most probable cause5. Determine if escalation is necessary6. Create an action plan and solution, identifying potential effects7. Implement and test the solution8. Identify the results and effects of the solution9. Document the solution and the entire process

Troubleshooting Troubleshooting ScenariosScenarios

• Troubleshooting Scenario One– “I can’t log in!”

• Biggest and most complex scenario• Log in from other machines• Attempt to log in yourself and try to PING• Use IPCONFIG or IFCONFIG

• Troubleshooting Scenario Two– “I can’t get to this Web site!”

• Have user try to reach another Web site• Try to ping the site by name and IP address• Determine whether DNS is functioning • Ping the gateway or proxy server • Ping other sites• If other sites reachable, notify problem site owner• If no sites available, notify your ISP

• Troubleshooting Scenario Three– “Our Web server is sluggish!”

• Connect from another location• If connection is also slow, problem with server• Check server logs for changes• Run Performance Monitor and compare new

logs with previously established baseline logs• If connection from other location OK, run

TRACERT command from user’s computer to reveal slow hop

• Troubleshooting Scenario Four– “I can’t see anything on the network!”

• Check connectivity• Ping a remote system• Ping loopback address

– If error, run ipconfig /all and fix settings– If no error, check hardware

» Run utility on NIC» Test cable

• It’s Time to Escalate!– Broadcast storms– Switching loops– Route problems– Routing loops– Proxy ARP

• Troubleshooting is fun!– Apply good troubleshooting methodology– Constantly increase your knowledge– Become a troubleshooting artist