COEN 355 WINTER 2016

NAME: Shreyas Srinivasa ReddyID NO: W1183161

NETWORK TROUBLESHOOTING TOLLS

AND TECHNIQUES

AUDIENCE:

The target audience for this report is Network Engineers. This report provides an insight into tools and techniques for troubleshooting the networks

Although the report will provide a brief overview of commands used for troubleshooting, the reader should have knowledge of network protocols.

1

Introduction.................................................................................................................................................4

Troubleshooting Overview......................................................................................................................4

Symptoms, Problems, and Solutions.......................................................................................................4

General Problem-Solving Model..............................................................................................................4

Preparing for Network Failure.................................................................................................................5

TOOLS OF THE TRADE..................................................................................................................................7

PING.........................................................................................................................................................7

ERROR INDICATION.............................................................................................................................7

ICMP....................................................................................................................................................8

ICMPv4 Echo (Request) and Echo Reply Messages..............................................................................8

Ping Drawbacks...................................................................................................................................8

TRACEROUTE...........................................................................................................................................8

Different types of Traceroute program................................................................................................9

Traceroute Drawbacks.......................................................................................................................10

Pathchar................................................................................................................................................10

Drawbacks.........................................................................................................................................11

bing....................................................................................................................................................12

Netcat....................................................................................................................................................13

Netcat Example:................................................................................................................................13

Netcat Drawbacks..............................................................................................................................14

Show Commands.......................................................................................................................................14

Show cam dynamic................................................................................................................................14

Show CDP neighbor...............................................................................................................................14

Cisco Discovery Protocol (CDP)..........................................................................................................14

Show ip route command.......................................................................................................................15

Show ip traffic........................................................................................................................................16

show ip cef.............................................................................................................................................17

2

Show process cpu..................................................................................................................................18

Show process mem................................................................................................................................18

Show system..........................................................................................................................................18

Debug Commands.................................................................................................................................18

debug ip packet detail...........................................................................................................................19

Debug IP Routing...................................................................................................................................19

Case Study: NetworkDown........................................................................................................................21

Problem statement:...............................................................................................................................21

How to deal this problem?....................................................................................................................21

Develop a plan of attack-.......................................................................................................................23

Future aspects and Conclusion..................................................................................................................24

Figure 1: ICMP ECHO_REQUEST/REPLAY.....................................................................................................8Figure 2:ICMP TTL exceeded messages.......................................................................................................9Figure 3:traceroute in cmd........................................................................................................................10Figure 4 Pathchar multilink........................................................................................................................10Figure 5:Bing link bandwidth.....................................................................................................................12Figure 6:HTTP Connectivity........................................................................................................................13Figure 7:Syslog Connectivity......................................................................................................................14Figure 8:show cdp neighbor......................................................................................................................15Figure 9: routers configured on OSPF........................................................................................................15Figure 10:debug ip packet detail...............................................................................................................19Figure 11:OSPF area 0................................................................................................................................19

3

Introduction

Troubleshooting Overview

Dependency on network resources has grown tremendously over the past ten years. In today's world, a company's success is highly dependent on its network availability. As a result, companies are increasingly less tolerant of network failures. Therefore, network troubleshooting has become a crucial element to many organizations.

Not only has the dependency for network grown, but the industry also is moving toward increasingly complex environments, involving multiple media types, multiple protocols, and often interconnection to unknown networks. These unknown networks may be defined as a transit network belonging to a Internet service provider (ISP), or a telco that interconnects private networks. The convergence of voice and video into data networks has also added to the complexity and the importance of network reliability. More complex network environments mean that the potential for connectivity and performance problems in internetworks is high, and the source of problems is often elusive.

Symptoms, Problems, and Solutions

Failures in internetworks are characterized by certain symptoms. These symptoms might be general (such as clients being incapable of accessing specific servers) or more specific (routes not existing in a routing table). Each symptom can be traced to one or more problems or causes by using specific troubleshooting tools and techniques. After being identified, each problem can be remedied by implementing a solution consisting of a series of actions.

General Problem-Solving Model

When you're troubleshooting a network environment, a systematic approach works best. An unsystematic approach to troubleshooting can result in wasting valuable time and resources, and can sometimes make symptoms even worse. Define the specific symptoms, identify all potential problems that could be causing the symptoms, and then systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear.

The following steps detail the problem-solving process:

Step 1 When analyzing a network problem, make a clear problem statement. You should define the problem in terms of a set of symptoms and potential causes.

To properly analyze the problem, identify the general symptoms and then ascertain what kinds of problems (causes) could result in these symptoms. For example, hosts might not be responding to service requests from clients (a symptom). Possible causes might include a misconfigured host, bad interface cards, or missing router configuration commands.

Step 2 Gather the facts that you need to help isolate possible causes.

Ask questions of affected users, network administrators, managers, and other key people. Collect information from sources such as network management systems, protocol analyzer traces, output from router diagnostic commands, or software release notes.

Step 3 Consider possible problems based on the facts that you gathered. Using the facts, you can eliminate some of the potential problems from your list.

Depending on the data, for example, you might be able to eliminate hardware as a problem so that you can focus on software problems. At every opportunity, try to narrow the number of potential problems so that you can create an efficient plan of action.

4

Step 4 Create an action plan based on the remaining potential problems. Begin with the most likely problem, and devise a plan in which only one variable is manipulated.

Changing only one variable at a time enables you to reproduce a given solution to a specific problem. If you alter more than one variable simultaneously, you might solve the problem, but identifying the specific change that eliminated the symptom becomes far more difficult and will not help you solve the same problem if it occurs in the future.

Step 5 Implement the action plan, performing each step carefully while testing to see whether the symptom disappears.

Step 6 Whenever you change a variable, be sure to gather results. Generally, you should use the same method of gathering facts that you used in Step 2 (that is, working with the key people affected, in conjunction with utilizing your diagnostic tools).

Step 7 Analyze the results to determine whether the problem has been resolved. If it has, then the process is complete.

Step 8 If the problem has not been resolved, you must create an action plan based on the next most likely problem in your list. Return to Step 4, change one variable at a time, and repeat the process until the problem is solved.

Preparing for Network Failure

It is always easier to recover from a network failure if you are prepared ahead of time. Possibly the most important requirement in any network environment is to have current and accurate information about that network available to the network support personnel at all times. Only with complete information can intelligent decisions be made about network change, and only with complete information can troubleshooting be done as quickly and as easily as possible.

During the process of network troubleshooting, the network is expected to exhibit abnormal behavior. Therefore, it is always a good practice to set up a maintenance time window for troubleshooting to minimize any business impact. Always document any changes being made so that it is easier to back out if troubleshooting has failed to identify the problem within the maintenance window.

To determine whether you are prepared for a network failure, answer the following questions:

•Do you have an accurate physical and logical map of your network?

Does your organization or department have an up-to-date network map that outlines the physical location of all the devices on the network and how they are connected, as well as a logical map of network addresses, network numbers, subnetworks, and so forth?

•Do you have a list of all network protocols implemented in your network?

For each of the protocols implemented, do you have a list of the network numbers, subnetworks, zones, areas, and so on that are associated with them?

•Do you know which protocols are being routed?

For each routed protocol, do you have correct, up-to-date router configuration?

•Do you know which protocols are being bridged?

Are any filters configured in any bridges, and do you have a copy of these configurations?

•Do you know all the points of contact to external networks, including any connections to the Internet?

5

For each external network connection, do you know what routing protocol is being used?

•Do you have an established baseline for your network?

Has your organization documented normal network behavior and performance at different times of the day so that you can compare the current problems with a baseline?

If you can answer yes to all questions, you will be able to recover from a failure more quickly and more easily than if you are not prepared. Lastly, for every problem solved, be sure to document the problems with solutions provided. This way, you will create a problem/answer database that others in your organization can refer to in case similar problems occur later. This will invariably reduce the time to troubleshoot your networks and, consequently, minimize your business impact.

6

TOOLS OF THE TRADE

PINGPing is a computer network administration software utility used to test the reach ability of a host on an Internet Protocol (IP) network and to measure the round-trip time for messages sent from the originating host to a destination computer and back. The name comes from active sonar terminology that sends a pulse of sound and listens for the echo to detect objects under water, however, the acronym "PING" meaning "Packet Inter Net Groper" has been in use since early days in computing for testing and measuring networks and the Internet.

Ping operates by sending Internet Control Message Protocol (ICMP) echo request packets to the target host and waiting for an ICMP echo reply. It measures the round-trip time from transmission to reception, reporting errors and packet loss. The results of the test usually include a statistical summary of the response packets received, including the minimum, maximum, the mean round-trip times, and usually standard deviation of the mean.

The command-line options for the ping utility and its output vary depending on implementation. Options may include the size of the payload, count of tests, limits for the number of hops (TTL) that probes traverse, and interval between the requests. Many systems provide a companion utility ping6, for similar testing on Internet Protocol version 6 (IPv6) networks.

ERROR INDICATION

In cases of no response from the target host, most implementations of ping display nothing, or periodically print notifications about timing out. Possible ping outputs indicating a problem include the following:

H, !N or !P – host, network or protocol unreachable S – source route failed F – fragmentation needed U or !W – destination network/host unknown I – source host is isolated A – communication with destination network administratively prohibited Z – communication with destination host administratively prohibited Q – for this ToS the destination network is unreachable T – for this ToS the destination host is unreachable X – communication administratively prohibited V – host precedence violation zC – precedence cutoff in effect

In case of error, the target host or an intermediate router sends back an ICMP error message, for example "host unreachable" or "TTL exceeded in transit". In addition, these messages include the first eight bytes of the original message (in this case header of the ICMP echo request, including the quench value), so the ping utility can match responses to originating queries.

$ ping -c 5 www.example.comPING www.example.com (93.184.216.119): 56 data bytes64 bytes from 93.184.216.119: icmp_seq=0 ttl=56 time=11.632 ms64 bytes from 93.184.216.119: icmp_seq=1 ttl=56 time=11.726 ms64 bytes from 93.184.216.119: icmp_seq=2 ttl=56 time=10.683 ms64 bytes from 93.184.216.119: icmp_seq=3 ttl=56 time=9.674 ms64 bytes from 93.184.216.119: icmp_seq=4 ttl=56 time=11.127 ms

--- www.example.com ping statistics ---5 packets transmitted, 5 packets received, 0.0% packet lossround-trip min/avg/max/stddev = 9.674/10.968/11.726/0.748 ms

Summarization of its results after completing the ping probes. The shortest round trip time was 9.674 ms, the average was 10.968 ms, and the maximum value was 11.726 ms. The measurement had a standard deviation of 0.748 ms.

7

ICMP

ICMP messages are sent in several situations: for example, when a datagram cannot reach its destination, when the gateway does not have the buffering capacity to forward a datagram, and when the gateway can direct the host to send traffic on a shorter route. The Internet Protocol is not designed to be absolutely reliable. The purpose of these control messages is to provide feedback about problems in the communication environment, not to make IP reliable. There are still no guarantees that a datagram will be delivered or a control message will be returned. Some datagrams may still be undelivered without any report of their loss. The higher level protocols that use IP must implement their own reliability procedures if reliable communication is required. The ICMP messages typically report errors in the processing of datagrams. To avoid the infinite regress of messages about messages etc., no ICMP messages are sent about ICMP messages.

ICMPv4 Echo (Request) and Echo Reply Messages 

One of the main purposes of ICMP informational messages is to enable testing anddiagnostics, to help identify and correct problems on an internetwork.The most basic test that can be conducted between two devices is simply checkingif they are capable of sending datagram’s to each other. The usual way that this isdone is to have one device send a test message to a second device,Which receives the message and replies back to tell the first device it received the message? ICMPv4 includes a pair of messages specifically for connection testing. Suppose Device A wants to see if it can reach Device B. Device A begins the test process by sending an ICMPv4 Echo message to B. Device B, when it receives the Echo, responds back to Device A with an Echo Reply message. When Device A receives this message, it knows that it is able to communicate (both send and receive) successfully with Device B.

Ping Drawbacks

Increases network load Uses artificially high TTL value Often routers lower the priority for ping to prevent DoS attacks Only does network- layer checks Does not pinpoint network problems

TRACEROUTE

Network engineers use this tool most commonly in their day to day activities. Its basically a network diagnostic tool that is very handy. There are three main primary objectives of traceroute tool. These objectives fulfilled by tracroute gives an insight to your network problem.

The entire path that a packet travels through Names and identity of routers and devices in your path Network Latency or more specifically the time taken to send and receive data to each devices on the path

Each IP packet that you send on the internet has got a field called as TTL. TTL stands for Time To Live. Although its called as Time To Live, its not actually the time in seconds, but its something else.TTL is not measured by the no of seconds but the no of hops. Its the maximum number of hops that a packet can travel through across the internet, before its discarded.

8

Figure 1: ICMP ECHO_REQUEST/REPLAY

What if there was no TTL at all?. If there was no TTL in an IP packet, the packet will flow endlessly from one router to another and on and on forever searching for the destination. TTL value is set by the sender inside IP packet. If the destination is not found after traveling through too many routers in between ( hops ) and TTL value becomes 0 (which means no further travel) the receiving router will drop the packet and informs the original sender.

The information send by the router receiving a packet with TTL of 1 back to the original sender is called as "ICMP TTL exceeded messages". Of course in internet when you send something to a receiver, the receiver will come to know the address of the sender.

Hence when an ICMP TTL exceeded message is sent by a router, the original sender will come to know the address of the router.

How trace route works?

Step 1: My Source address will make a packet with destination ip address of 8.8.8.8 and a destination port number between 33434 to 33534. And the important thing it does it to make the TTL Value 1 Step 2: Of course my packet will reach my gateway server. On seeing receiving the packet my gateway server will reduce the TTL by 1 (All routers/hops in between does this job of reducing the TTL value by 1). Once the TTL is reduced by the value of 1 (1-1= 0), the TTL value becomes zero. Hence my gateway server will send me back a TTL Time exceeded message. Please remember that when my gateway server sends a TTL exceeded message back to me, it will send the first 28 byte header of the initial packet i send.

Step 3: On receiving this TTL Time exceeded message, my traceroute program will come to know the source address and other details about the first hop (Which is my gateway server.). Step 4: Now the traceroute program will again send the same UDP packet with the destination of 8.8.8.8, and a random UDP destination port between 33434 to 33534. But this time i will make the initial TTL 2. This is because my gateway router will reduce it by 1 and then forwards that same packet which send to the next hop/router (the packet send by my gateway to its next hop will have a TTL value of 1).

Step 5: On receiving UDP packet, the next hop to my gateway server will once again reduce it to 1 which means now the TTL has once again become 0. Hence it will send me back a ICMP Time exceeded message with its source address, and also the first 28 byte header of the packet which i send. Step 6: On receiving that message of TTL Time Exceeded, my traceroute program will come to know about that hop/routers IP address and it will show that on my screen. Step 7: Now again my traceroute program will make a similar UDP packet with again a random udp port with the destination address of 8.8.8.8. But this time the ttl value is made to 3, so that the ttl will automatically become 0, when it reaches the third hop/router(Please remember that my gateway and the next hop to it, will reduce it by 1 ). So that it will reply me with a TTL Time exceeded message, and my traceroute program will come to know about that hop/routers IP address.Step 8: On receiving that reply, the traceroute program will once again make a UDP packet with TTL value of 4 this time. If i

gets a TTL Time exceeded for that also, then my traceroute program will send a UDP packet with TTL of 5 and so on.

Different types of Traceroute program

There are different types of traceroute programs. Each of them works slightly differently. But the overall concept behind each of them is the same. All of them uses the TTL value.Why different implementations? That because you can use the one which is applicable to your environment. If suppose A firewall block the UDP traffic then you can use another traceroute for this purpose. The different types are mentioned below.

• UDP Traceroute• ICMP traceroute• TCP Traceroute

9

Figure 2:ICMP TTL exceeded messages

Traceroute Drawbacks

• ICMP messages may be filtered

• Different IP stacks respond differently to traceroute

• Latency figures may not be accurate with regard to applications

Pathchar

One tool that automates this process is pathchar. This tool, written by Van Jacobson several years ago, seems to be in a state of limbo. It has, for several years, been available as an alpha release, but nothing seems to have been released since. Several sets of notes or draft notes are available on the Web, but there appears to be no man page for the program. Nonetheless, the program remains available and has been ported to several platforms. Fortunately, a couple of alternative implementations of the program have recently become available. These include Bing, pchar, clink, and tmetric.

One strength of pathchar and its variants is that they can discover the bandwidth of each link along a path using software at only one end of the path. The method used is basically that described earlier for ping, but pathchar uses a large number of packets of various sizes. Here is an example of running  pathchar.

0: 10.29.100.33 (nms-server2.rtp.cisco.com)Partial loss: 1 / 1472 (0%)Partial char: rtt = 1.872604 ms, (b = 0.000876 ms/B), r2 = 0.998560

stddev rtt = 0.004053, stddev b = 0.000005Partial queuing: avg = 0.001308 ms (1492 bytes)Hop char: rtt = 1.872604 ms, bw = 9130.730297 KbpsHop queuing: avg = 0.001308 ms (1492 bytes)

Path length: 4 hopsPath char: rtt = 4.692853 ms r2 = 0.999996Path bottleneck: 126.277240 KbpsPath pipe: 74 bytesPath queuing: average = 0.003377 ms (1582 bytes)

 pathchar runs, it first displays a message describing how the probing will be done. From output, we see that pathchar is using different packet sizes ranging from 64 to 1500 bytes. (1500 is the local host's MTU.) It uses 32 different sets of these packets for each hop. Thus, this eight-hop run generated 11,520 test packets plus an equal number of replies.

The bandwidth and delay for each link is given. pathchar may also include information on the queuing delay. Pathchar is not always successful in estimating the bandwidth or the delay .

As pathchar runs, it shows a countdown as it sends out each packet. It will display a line that looks something like this:

1: 31 288 0 3

The 1: refers to the hop count and will be incremented for each successive hop on the path. The next number counts down, giving the number of sets of probes remaining to be run for this link. The third number is the size of the current packet being sent. Both the second and third numbers should be changing rapidly. The last two numbers give the number of packets that have been dropped so far on this link and the average round-trip time for this link.

10

Figure 3:traceroute in cmd

Figure 4 Pathchar multilink

When the probes for a hop are complete, this line is replaced with a line giving the bandwidth, incremental propagation delay, and round-trip time. pathchar uses the minimum of the observed delays to improve its estimate of bandwidth.

Several options are available with pathchar. Of greatest interest are those that control the number and size of the probe packet used. The option -q allows the user to specify the number of sets of packets to send. The options -m and -M control the minimum and maximum packet sizes, respectively. The option -Q controls the step size from the smallest to largest packet sizes. As a general rule of thumb, more packets are required for greater accuracy, particularly on busy links. The option -nturns off DNS resolution, and the option -v provides for more output.

Pathchar is not without problems. One problem for pathchar is hidden or unknown transmission points. The first link reports a bandwidth of 4.3 Mbps. Fromtraceroute, we only know of the host and the router at the end of the link. This is actually a path across a switched LAN with three segments and two additional transmission points at the switches. The packet is transmitted onto a 10-Mbps network, then onto a 100-Mbps backbone, and then back onto a 10 Mbps network before reaching the first router. Consequently, there are three sets of transmission delays rather than just one, and a smaller than expected bandwidth is reported.

You will see this problem with store-and-forward switches, but it is not appreciable with cut-through switches. (the sidebar "Types of Switches" if you are unfamiliar with the difference between cut-through and store-and-forward switches.) In a test in which another switch, configured for cut-through, was added to this network, almost no change was seen in the estimated bandwidth with pathchar. When the switch was reconfigured as a store-and-forward switch, the reported bandwidth on the first link dropped to 3.0 Mbps.

Types of Switches

Devices may minimize queuing delays by forwarding frames as soon as possible. In some cases, a device may begin retransmitting a frame before it has finished receiving that frame. With Ethernet frames, for example, the destination address is the first field in the header. Once this has been read, the out interface is known and transmission can begin even though much of the original frame is still being received. Devices that use this scheme are called cut-through devices.

The alternative is to wait until the entire frame has arrived before retransmitting it. Switches that use this approach are known as store-and-forwarddevices.

Cut-through devices have faster throughput than store-and-forward switches because they begin retransmitting sooner. Unfortunately, cut-through devices may forward damaged frames, frames that a store-and-forward switch would have discarded. The problem is that the damage may not be discovered by the cut-through device until after retransmission has already begun. Store-and-forward devices introduce longer delays but are less likely to transmit damaged frames since they can examine the entire frame before retransmitting it. Store-and-forward technology is also required if interfaces operate at different speeds. Often devices can be configured to operate in either mode.

This creates a problem if you are evaluating an ISP. For example, it might appear that the fourth link is too slow if the contract specifies T1 service. This might be the case, but it could just be a case of a hidden transmission point. Without more information,

this isn't clear.

Drawbacks

• ICMP messages may be filtered

• Latency figures may not be accurate with regard to applications

• Different IP stacks respond differently to pchar

11

bing

One alternative to pathchar is bing, a program written by Pierre Beyssac. Where pathchar gives the bandwidth for every link along a path, bing is designed to measure point-to-point bandwidth. Typically, you would run traceroute first if you don't already know the links along a path. Then you would run bing specifying the near and far ends of the link of interest on the command line. This example measures the bandwidth of the third hop

bsd1# bing -e10 -c1 205.153.60.2 165.166.36.17BING 205.153.60.2 (205.153.60.2) and 165.166.36.17 (165.166.36.17) 44 and 108 data bytes1024 bits in 0.835ms: 1226347bps, 0.000815ms per bit1024 bits in 0.671ms: 1526080bps, 0.000655ms per bit1024 bits in 0.664ms: 1542169bps, 0.000648ms per bit1024 bits in 0.658ms: 1556231bps, 0.000643ms per bit1024 bits in 0.627ms: 1633174bps, 0.000612ms per bit1024 bits in 0.682ms: 1501466bps, 0.000666ms per bit 1024 bits in 0.685ms: 1494891bps, 0.000669ms per bit1024 bits in 0.605ms: 1692562bps, 0.000591ms per bit1024 bits in 0.618ms: 1656958bps, 0.000604ms per bit

--- 205.153.60.2 statistics ---bytes out in dup loss rtt (ms): min avg max 44 10 10 0% 3.385 3.421 3.551 108 10 10 0% 3.638 3.684 3.762

--- 165.166.36.17 statistics ---bytes out in dup loss rtt (ms): min avg max 44 10 10 0% 3.926 3.986 4.050 108 10 10 0% 4.797 4.918 4.986

--- estimated link characteristics ---estimated throughput 1656958bpsminimum delay per packet 0.116ms (192 bits)average statistics (experimental) :packet loss: small 0%, big 0%, total 0%average throughput 1528358bpsaverage delay per packet 0.140ms (232 bits)weighted average throughput 1528358bps

resetting after 10 samples.

The output begins with the addresses and packet sizes followed by lines for each pair of probes. Next, bing returns round-trip times and packet loss data. Finally, it returns several estimates of throughput. The observant reader will notice that bing reported throughput, not bandwidth. Unfortunately, there is a lot of ambiguity and inconsistency surrounding these terms.

In this particular example, we have specified the options -e10 and -c1, which limit the probe to one cycle using 10 pairs of packets. Alternatively, you can omit these options and watch the output. When the process seems to have stabilized, enter a Ctrl-C to terminate the program. The summary results will then be printed. Interpretation of these results should be self-explanatory.

bing allows for a number of fairly standard options. These options allow controlling the number of packet sizes, suppressing name resolution, controlling routing, and obtaining verbose output. See the manpage if you have need of these options.

Because bing uses the same mechanism as pathchar, it will suffer the same problems with hidden transmission points. Thus, you should be circumspect when using it if you don't fully understand the topology of the network. While bing does not generate nearly as much traffic as pathchar, it can still place strains on a network.

12

Figure 5:Bing link bandwidth

Netcat

Netcat is similar in operation to telnet. It tests application connectivity and tests TCP and UDP services. Netcat (often abbreviated to nc) is a computer networking utility for reading from and writing to network connections using TCP orUDP. Netcat is designed to be a dependable back-end that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and investigation tool, since it can produce almost any kind of connection its user could need and has a number of built-in capabilities.

Its list of features includes port scanning, transferring files, and port listening, and it can be used as a backdoor.

The original netcat's features include:

Outbound or inbound connections, TCP or UDP, to or from any ports Full DNS forward/reverse checking, with appropriate warnings Ability to use any local source port Ability to use any locally configured network source address Built-in port-scanning capabilities, with randomization Built-in loose source-routing capability Can read command line arguments from standard input Slow-send mode, one line every N seconds Hex dump of transmitted and received data Optional ability to let another program service establish connections Optional telnet-options responder Featured tunneling mode which permits user-defined tunneling, e.g., UDP or TCP, with the possibility of specifying all

network parameters (source port/interface, listening port/interface, and the remote host allowed to connect to the tunnel).

Rewrites like GNU's and OpenBSD's support additional features. For example, OpenBSD's nc supports TLS.

Netcat Example:

Verify HTTP Connectivity

% nc –v –w 3 nms-server2.cisco.com 80 nms-server2.cisco.com [172.18.124.33] 80 (http)

open GET / HTTP/1.0

HTTP/1.1 200 OK [HTML data]

Verify Syslog Connectivity

%echo ‘<38>Hello World’ | nc –w 1 –u nms-server2 514

%tail –1 /var/log/messages

Apr 17 00:54:45 nms-server2 Hello World

13

Figure 6:HTTP Connectivity

Figure 7:Syslog Connectivity

Netcat Drawbacks

• Does not measure network performance

• Does not attempt to isolate where the connectivity problem lies in the network

Show Commands

Show cam dynamic

This command describes how to collect Dynamic Content-Addressable Memory (CAM) entries for Catalyst switches using Simple Network Management Protocol (SNMP). It’s a Cisco Catalyst OS command which shows MAC to port mapping .The BRIDGE-MIB and Q-BRIDGE-MIB can be used to obtain this information via SNMP

* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry. X = Port

Security Entry

VLANDest MAC/Route Des [CoS]

Destination Ports or VCs / [Protocol Type]

---------------------- -----

-------------------------------------------

18 00-10-0d-38-10-00 5/3 [ALL]6 00-30-94-1c-46-ff 5/3 [ALL]100 00-90-27-86-76-e2 5/1 [ALL]18 00-00-0c-07-ac-12 5/3 [ALL]100 00-04-de-a9-18-00 5/3 [ALL]6 00-04-4e-f2-c8-00 5/3 [ALL]19 00-10-0d-a1-18-80 5/3 [ALL]

Show CDP neighbor

Cisco Discovery Protocol (CDP)

CDP uses Layer 2 multicast for advertisements. It also uses special multicast MAC address so that Cisco devices will not forward CDP packets. It basically runs on virtually all Cisco devices. CDP is enabled by default on all broadcast interfaces and displays information about directly connected neighbors. Useful for debugging connectivity issues as well as building topology maps. The CISCO-CDP-MIB can be used to obtain CDP information via SNMP

It must be Configured only on links between Cisco devices, Do not configure CDP on links you do not manage

The following is a sample of output from the show cdp neighbors command. Device ID, interface type and number, holdtime settings, capabilities, platform, and port ID information about neighbors is displayed.

14

nms-3640a#show cdp neighbor

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater

Device ID Local Intrfce Holdtme Capability Platform Port ID

009042675(nms-500Eth 0/0 152 T B SWS-C5000 3/5

nms-2610a.rtp.cisSer 3/0 169 R 2610 Ser 1/0

The following is a sample of output for one neighbor from the show cdp neighbors detail   command. Additional detail is shown about neighbors including network address, enabled protocols, and software version.

ID: 009042675(nms-5000b)

Entry address(es):

IP address: 14.32.4.10Platform: WS-C5000, Capabilities: Trans-Bridge Source-Route-Bridge Switch

Interface: Ethernet0/0,Port (outgoingport):3/5Holdtime : 174 sec

Version :WS-C5000 Software, Version McpSW: 5.4(4) NmpSW: 5.4(4)Copyright (c) 1995-2000 by Cisco Systemsadvertisement version: 2VTP Management Domain: 'nms-remote' Native VLAN: 4Duplex: full

Show ip route command

One of the most important skills to have when it comes to basic networking is the ability to look at a routing table and determine exactly where a packet will be routed when it comes to a router. Sometimes a routing table is relatively simple, and this process is easy. However, many times this is not the case. In large networks, especially networks that implement a hub and spoke design where core routers are often required to know hundreds of routes or more, this can be tedious. I’d like to discuss the routing table on a Cisco router, and identify a few things to look for when trying to identify routing configuration.

All routers were simply configured to run OSPF on all links, and in one OSPF area. Take a look at the output of the “show ip route” command, issued on R0:

R0#show ip route

.....

1.0.0.0/30 is subnetted, 4 subnets

C    1.1.1.0 is directly connected, FastEthernet0/1

O    1.1.1.4 [110/2] via 1.1.1.2, 00:10:04, FastEthernet0/1

O    1.1.1.8 [110/2] via 1.1.1.13, 00:10:04, FastEthernet0/0

C    1.1.1.12 is directly connected, FastEthernet0/0

172.16.0.0/24 is subnetted, 4 subnets

C    172.16.0.0 is directly connected, Ethernet0/0/0

O    172.16.1.0 [110/11] via 1.1.1.2, 00:10:04, FastEthernet0/1

O    172.16.2.0 [110/12] via 1.1.1.13, 00:09:24, FastEthernet0/0

[110/12] via 1.1.1.2, 00:09:24, FastEthernet0/1

O    172.16.3.0 [110/11] via 1.1.1.13, 00:10:04, FastEthernet0/0

15

Figure 8:show cdp neighbor

Figure 9: routers configured on OSPF

“Directly connected”, represented by the letter C. These are routes that were not distributed via OSPF, but are networks that are directly connected to the router. The router inherently knows of these networks, so there’s no need to hear about them from another router. The other routes are represented by an “O”, which identifies them as routes received via an OSPF neighbor relationship.

O 172.16.2.0 [110/12] via 1.1.1.13, 00:09:24, FastEthernet0/0

[110/12] via 1.1.1.2, 00:09:24, FastEthernet0/1

Based on this specific route, the packets from PC1 will be sent to R0, then to either R1 or R3. We know this because these are the routers that use the next-hop addresses specified in the route. In addition, both the administrative distance values and the metrics are the same. This is because both routes were obtained through OSPF, and all links in question have the same cost.

These packets require one and only one route to get to the destination, so which one is used? Lets run a traceroute on PC1 to PC2 to and see which route this traffic takes

So it looks like the first hop after R0 is Lets run that trace again, with no changes to our routing configuration:

1.1.1.13, which is R3. We now have a different next-hop after R0. I Instead, the next hop is now R1.

The reason for this is that OSPF is performing automatic load balancing on this route. Since the two next-hops for the route to 172.16.2.0/24 from R0’s perspective were essentially tied, R0 is load balancing traffic between the two possible routes. This gets into a bit more advanced topics concerning OSPF, so I’ll save that for another day.

In the meantime, we’ve learned how a routing table works, and how routes are decided. In the event of a tie, there are certain measures, some manual, some automatic (as we’ve seen with OSPF) that can be used to break the tie.

Show ip traffic

It mainly focuses on error trends and identify types of errors; for example input errors usually indicate faulty hardware. This command displays IP traffic summaries for the IP, ICMP, UDP, TCP protocols. Use the show ip traffic command with no arguments to display the IP traffic counts for all slots. Use the show ip traffic module bay#/slot# to display the IP traffic counts for a specified slot.

IP statistics:713345 local destinationRcvd: 8586090 total,

0 format errors, 0 checksum errors, 1764 bad hop count0 unknown protocol, 0 not a gateway

0 with optionsOpts:

0 security failures, 0 bad options,0 end, 0 nop, 0 basic security, 0 loose source route0 timestamp, 0 extended security, 0 record route0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

Frags:0 other

0 timeouts, 0 couldn't reassemble0 reassembled,

Bcast:0 fragmented, 0 couldn't fragment2260 received, 0 sent

Mcast: 351508 received, 87579 sentSent: 475639 generated, 7818883 forwardedDrop: 60 encapsulation failed, 0 unresolved, 0 no adjacency

0 no route, 0 unicast RPF, 0 forced drop

16

Drop:

0 options denied

0 packets with source IP address zero

show ip cef

This command functions somewhat like show ip route, but shows information from the forwarding plane itself (the FIB instead of the RIB). As such, its output is rather spartan and to the point. Verify next hops and interfaces are correct for given route prefixes. Corrupted CEF tables can cause strange routing behaviors. CEF output includes a few entries which don't appear in show ip route, such as the default route to null0.

Router# show ip route 10.0.9.8Routing entry for 10.0.9.8/30 Known via "ospf 1", distance 110, metric 11, type intra area Last update from 10.0.9.2 on FastEthernet0/1, 00:09:33 ago Routing Descriptor Blocks: * 10.0.9.2, from 10.0.0.4, 00:09:33 ago, via FastEthernet0/1 Route metric is 11, traffic share count is 1

Router# show ip cef 10.0.9.810.0.9.8/30, version 20, epoch 0, cached adjacency 10.0.9.20 packets, 0 bytes tag information set local tag: 17 via 10.0.9.2, FastEthernet0/1, 0 dependencies next hop 10.0.9.2, FastEthernet0/1 valid cached adjacency tag rewrite with Fa0/1, 10.0.9.2, tags imposed: {}

We can filter the routes we want to see by specifying a network and mask and then appending the longer-prefixes keyword. For example, if we only wanted to see routes within 10.0.0.0/24.

Router# show ip cef 10.0.0.0 255.255.255.0%Prefix not foundRouter# show ip cef 10.0.0.0 255.255.255.0 longer-prefixesPrefix Next Hop Interface10.0.0.1/32 receive10.0.0.2/32 10.0.9.2 FastEthernet0/110.0.0.3/32 10.0.9.6 FastEthernet1/010.0.0.4/32 10.0.9.2 FastEthernet0/110.0.0.5/32 10.0.9.13 FastEthernet0/0

Suppose if we want to see all routes which point out a given interface. Instead of trying to glean this information from  show ip route, you can specify an interface with show ip cef interface

Router# show ip cef f0/0Prefix Next Hop Interface10.0.0.5/32 10.0.9.13 FastEthernet0/010.0.9.12/30 attached FastEthernet0/010.0.9.13/32 10.0.9.13 FastEthernet0/010.0.9.16/30 10.0.9.13 FastEthernet0/0 10.0.9.2 FastEthernet0/1

We can also view routes of a specific CEF adjacency type.

Router# show ip cef adjacency dropPrefix Next Hop Interface224.0.0.0/4 dropRouter# show ip cef adjacency gleanPrefix Next Hop Interface10.0.9.0/30 attached FastEthernet0/110.0.9.4/30 attached FastEthernet1/010.0.9.12/30 attached FastEthernet0/0

We can also be used to predict the route of an explicit source and destination address pair. This can be handy when equal-cost load balancing or source-based policy routing is in place.

17

Router# show ip cef exact-route 10.0.9.5 192.168.0.110.0.9.5 -> 192.168.0.1 : FastEthernet0/1 (next hop 10.0.9.18)Router# show ip cef exact-route 10.0.9.6 192.168.0.110.0.9.6 -> 192.168.0.1 : FastEthernet0/0 (next hop 10.0.9.14)

Show process cpu

Check to make sure overall system load is under control. Use the process list to determine which process might be misbehaving.

CPU utilization for five seconds: 29%/6%; one minute: 8%; five minutes: 5%PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 880 1823462 0 0.00% 0.00% 0.00% 0 Load Meter2 572128 2351401 243 0.00% 0.00% 0.00% 0 OSPF Hello3 0 106 0 0.00% 0.00% 0.00% 0 RTR Scheduler4 6118648 1117174 5476 0.00% 0.08% 0.10% 0 Check heaps5 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager6 0 2 0 0.00% 0.00% 0.00% 0 Pool Manager7 0 2 0 0.00% 0.00% 0.00% 0 Timers8 0 36 0 0.00% 0.00% 0.00% 0 Serial Backgroun9 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

10 1832 112 16357 30.01% 10.03% 7.44% 18 Virtual Exec

Show process mem

Show process mem checks to make sure no process is hogging memory. Use the process list to determine which process might be misbehaving .

Total: 34074032, Used: 6937220, Free: 27136812PID TTY Allocated Freed Holding Getbufs Retbufs Process

1 0 0 0 6868 0 0 Chunk Manager2 0 188 188 3868 0 0 Load Meter3 0 12120 2512624 10980 780 320 OSPF Hello4 0 0 0 6868 0 0 Check heaps5 0 1544 0 8412 0 0 Chunk Manager6 0 9896152 6360264 6868 3092496 3772896 Pool Manager7 0 188 188 6868 0 0 Timers8 0 222880 137936 6868 0 56000 Serial Backgroun9 0 0 0 6868 0 0 ALARM_TRIGGER_SC

10 0 1104690216 1544572760 29212 0 0 SNMP ENGINE

Show system

It’s a Cisco Catalyst OS command which shows environmental stats as well as peak and current traffic load.

PS1-Status PS2-Status---------- ----------ok none

Fan-Status Temp-Alarm Sys-Status Uptime d,h:m:s Logout---------- ---------- ---------- -------------- ---------ok off ok 7,07:46:16 20 minPS1-Type PS2-Type-------------------- --------------------WS-CAC-1000W none

Modem Baud Traffic Peak Peak-Time------- ----- ------- ---- -------------------------disable 9600 5% 7% Thu May 3 2001, 10:06:00

PS1 Capacity: 852.60 Watts (20.30 Amps @42V)

System Name System Location System Contact CC------------------------ ------------------------ ------------------------ ---

"NMS Pod"

18

Debug Commands

Use debug commands with caution. In general, it is recommended that these commands only be used under the direction of your

router technical support representative when troubleshooting specific problems. Enabling debugging can disrupt operation of the

router when internetworks are experiencing high load conditions. Hence, if logging is enabled, the access server can

intermittently freeze as soon as the console port gets overloaded with log messages.

debug ip packet detail 

The debug ip packet detail command, we have the option to enter the name or number of an access list. Doing that causes the debug command to get focused only on those packets satisfying (permitted by) the access list's statements. Here is an example. Imagine that host A has trouble making a Telnet connection to host B, and you decide to use debug on the router connecting the segments where hosts A and B reside Considering the addressing scheme, the access list 100 permits TCP traffic from host A (10.1.1.1) to host B (172.16.2.2) with the Telnet port (23) as the destination. Access list 100 also permits established TCP traffic from host B to host A. Using access list 100 with the debug ip packet detail command (as shown in the figure) allows you to see only debug packets that satisfy the access list.

This is an effective troubleshooting technique that requires less overhead on your router, while allowing all information on the subject you are troubleshooting to be displayed by the debug facility.

Debug IP Routing

There are two points of view on if this is a command. The plus side is that when you do a redistribution command you can instantly see if you are getting any feedback or not, as you can easily see routes leaving and entering the routing table.The negative argument is that if you fully understand what you are doing with your redistribution and can interpret the show ip route commands you shouldn’t need to be running debug ip routing. For some it is like a reassurance that everything is stable.

Lets take this example below with 3 routers running OSPF area 0

R3#sh ip routeCodes: C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS lev ia - IS-IS inter area, * - candidate default, U - per-user static o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

3.0.0.0/32 is subnetted, 1 subnetsC 3.3.3.3 is directly connected, Loopback0 10.0.0.0/24 is subnetted, 2 subnetsO 10.0.0.0 [110/20] via 10.0.1.2, 00:00:19, FastEthernet0/0C 10.0.1.0 is directly connected, FastEthernet0/0R3#

19

Figure 10:debug ip packet detail

Figure 11:OSPF area 0

Now we are going to enable the debug ip routing and add Loopback0 on Router 1 into OSPF area 0 – what we should see is the route coming into ospf

R3#debug ip routingIP routing debugging is onR3#

Now hop over to Router 1 and bring Loopback0 into ospf

R1(config)#int lo0R1(config-if)#ip ospf 1 area 0R1(config-if)#

Now we are going back to R3 and hopefully we should see the 1.1.1.1/32 get added into OSPF – as you can see it does get added, I then removed the interface from ospf and you can see that the prefix was then deleted.

R3#*Mar 1 00:30:48.247: RT: add 1.1.1.1/32 via 10.0.1.2, ospf metric [110/21]*Mar 1 00:30:48.251: RT: NET-RED 1.1.1.1/32R3#*Mar 1 00:35:01.703: RT: del 1.1.1.1/32 via 10.0.1.2, ospf metric [110/21]*Mar 1 00:35:01.707: RT: delete subnet route to 1.1.1.1/32*Mar 1 00:35:01.707: RT: NET-RED 1.1.1.1/32*Mar 1 00:35:01.707: RT: delete network route to 1.0.0.0*Mar 1 00:35:01.707: RT: NET-RED 1.0.0.0/8

This is the kind of output you would expect to see of  a route being learned and then removed, learned and then removed. This would indicate a possible routing loop. This command is great for learning how redistribution works and monitoring routes going into the routing table.

20

Case Study: NetworkDown

Problem statement: Inaccessibility of internet, reading mails or using applications

How to deal this problem?First and foremost important thing to be done in order to deal with any problem is to know its roots or background so first know the network topology before digging into the issue.

In order to know what the issue is make sure the right questions are asked such that all possibilities are covered. First in the list of questions is WHAT – What other users are seeing the problem? What users are not seeing the problem? All the information should be documented in the proper format so that no information is missed out. Here, table structure best suits and is as shown below:

Now both type of users who faced the problem and who did not come across should be tracked to get the information on WHERE exactly the problem is caused. For this, CiscoWorks provides the UserTracking facility based on the username. This User Tracking mechanism provides information like MACAddress, DeviceName etc., VLAN column gives information on what VLAN is utilized by each user and its status. For example, here the user cathy is not facing any issue and thus the VLAN is inactive999. Sam and tom are facing the problem and thus VLAN data is entered.

21

Then gather information about WHEN - When was the problem faced first? When did you see this problem in the past? Now the document should be as below after the information is collected by the users.

Last question to be answered is HOW MUCH - Since the problem started, when has the network operated correctly? If it never operated correctly since the problem began then it will be 100% and sporadic when it doesn’t remain regular at all times.

As all requirements are collected, analysis can be started. Difference between the users who faced the problem (sam, tom) and not faced (cathy) should be analyzed. Variances within these users are found to be Vlan, Subnet and Spantree. In particular Vlan124 is used by users who are facing issue but cathy is

22

using Vlan999. So differences between these two Vlan is evaluated and is found to be VTP Domain, VTP Server and Spantree. All these are noted into the document.

Changes made before the problem occurrence is tracked with the help of both Traps and Syslog. Concentrating more towards the Vlan124, topology services provides the information on spanning tree. Current spanning tree is compared with the spanning tree which existed before.

Develop a plan of attack- based on all the information collected so far, root cause for network outage is assumed to be spanning tree loop. To confirm the immediate cause, further analysis on topology and network statistics are done and compared to good baseline. Document your actions and your analysis on the same in the form of Action and Notes columns. To confirm immediate cause, time tab is utilized and before and after issue snapshots are compared.

To know what caused this Spanning Tree Loop, configuration changes made are tracked and it was found that Fa0/48 was half duplex and then it was rectified by using below hardcodes.

23

The problem is solved. Port duplex is consistent on both sides of the link, the la-surf port is no longer resetting due to excessive errors; the network has been restored to its previous working state, and all users have connectivity again

Future aspects and Conclusion

Network downtime, to some extent, is unavoidable. No matter how well prepared you may be or how stable your network is, at some point you’re likely to deal with an incapacitated server, or an unreachable network segment. What matters most, though, is how you respond to a network outage, and how quickly you can get back on your feet. Being prepared for the worst is crucial when running a large-scale environment – pre-empting network issues means you’ll experience far less downtime, and when you do lose connectivity, you’ll be able to recover in a fraction of the time, while minimizing the impact to productivity and profit.

1. Choose your weapon: For most small to medium sized businesses, a single server with monitoring software should be enough to handle all of your monitoring needs. However, if your network is distributed among several sites, it may help to use more than one server. Most monitoring suites will allow client or browser-based remote access, so it’s worth considering having an off-site monitoring station. This will ensure that you don’t have a single point of failure, and also means you’ll know when there is an internet connection failure or a disruption to your local network.

2. Out with the old: It may sound like a no-brainer, but old network hardware can be the cause of a host of network troubleshooting problems. Aging server equipment can cause unexpected downtime on your network at best, and potentially lead to data loss at worst. The initial cost of replacing old hardware might seem significant, but the investment will pay itself off several times over.

3. Know your devices: Knowing which devices are connected to your network at which times, is key to  determining how your network should run. While ping tests can return basic operational information such as traffic, packet errors and discards, consider using specialized plug-in or queries to collect more particular types of data. Remember to adjust accordingly when you hire new staff, upgrade company hardware or purchase new devices for your team.

4. Always Take Backup: It shouldn’t be news to anybody, but it’s a sad truth that many network troubleshooting issues can be avoided by having a rigorous backup schedule in place. Aside from the obvious loss of hours on network downtime, failing to back up important data could leave you to deal with hours of wasted work, disgruntled employees, and a lot of overtime salaries. Cloud-based storage is the perfect solution for being able to back up data on-the go from anywhere, so consider investing in cloud storage for your backup.

5. Schedule & Report : Using network monitoring software doesn’t mean that you can sit back and let the computer do all the work – you’ll still need to be smart about your scheduling and reporting so that you get the most out of your software. Make sure that your monitoring schedule takes into account the different levels of activity your network is likely to experience on an average day, and keep your network troubleshooting reports concise and easy to understand. Having a graphic information display, for instance, will help your networking team determine the environment’s status at a glance. Visualization of complex network data makes for less tedious troubleshooting when complex network issues occur.

6. Know what you’re working with: There are many ways to go about monitoring your network. Your network troubleshooting strategy should be informed by the business’s demand for service availability and uptime, as well as external factors such as the size of your business, the number of devices connected and how busy you are at any given

24

moment. It’s always helpful to measure the amount of used versus unused resources in your network in addition to its performance.

References

http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1901.pdf https://www.google.com/search?

q=network+troubleshooting&safe=off&espv=2&biw=1366&bih=643&source=lnms&tbm=isch&sa=X&ved=0ahUKEwixs-aSqYXLAhVK6mMKHT3EACYQ_AUIBygC#imgrc=4dPQI0wd_9ODPM%3A

http://www.irisns.com/six-proactive-monitoring-tips-to-reduce-network-troubleshooting- time/

https://www.google.com/search? q=bing&safe=off&source=lnms&tbm=isch&sa=X&ved=0ahUKEwi7zsqV_ITLAhUM_WMKHUpxDAwQ_AUICigE&biw=1366&bih=643#safe=off&tbm=isch&q=pathchar&imgrc=Vx3kqY_5LSQQaM%3A

http://www.cisco.com/c/en/us/td/docs/ios/ipapp/command/reference/iap_s3.html http://www.techrepublic.com/blog/data-center/10-commands-you-should-master-when-

working-with-the-cisco-ios-104071/ http://www.cisco.com/c/en/us/support/docs/ip/simple-network-management-protocol-

snmp/13492-cam-snmp.html?referring_site=RE&pos=1&page=http://www.cisco.com/c/en/us/support/docs/ip/simple-network-management-protocol-snmp/44800-mactoport44800.html

25

ISP - Internet service provider TELCO - Telecommunications company, provides telecommunications services such as telephony and data communicationsICMP - Internet Control Message Protocol TTL – Time to liveIPv6 - Internet Protocol version 6  PING - Packet Inter Net GroperIP - Internet Protocol ICMPv4 - Internet Protocol version 4UDP – User datagram protocol TCP – Transmission control protocolDNS – Domain name server Nc NetcatBSD - Berkeley Software Distribution TLS - Transport Layer SecurityHTTP – Hyper text transfer ProtocolSyslogCAM - Content-Addressable Memory SNMP - Simple Network Management Protocol MAC- media access controlBRIDGE-MIB and Q-BRIDGE-MIB - MODULE-IDENTITYCDP - Cisco Discovery Protocol O - OSPF - Open Shortest Path FirstR – RIP -  Routing Information ProtocolIS-IS- Intermediate System to Intermediate System C - connected, S - static, R – RIP -  Routing Information ProtocolB – BGP – Border gateway protocol D – EIGRP - Enhanced Interior Gateway Routing Protocol VLAN - virtual local area network

26