Catalyst 6500/6000 Switch High CPU Utilization Document ID: 63992 Introduction Prerequisites Requirements Components Used Conventions Difference Between CatOS and Cisco IOS System Software Understand CPU Utilization on Catalyst 6500/6000 Switches Situations and Features That Trigger Traffic to Go to Software Packets That Are Destined to the Switch Packets and Conditions That Require Special Processing ACL-Based Features NetFlow-Based Features Multicast Traffic Other Features IPv6 Situations LCP Schedular and DFC Module Common Causes and Solutions for High CPU Utilization Issues IP Unreachables NAT Translations Use of CEF FIB Table Space in the Flow Cache Table Optimized ACL Logging Rate Limit of Packets to the CPU Physical Merger of VLANs Due to Incorrect Cabling Broadcast Storm BGP Next-Hop Address Tracking (BGP Scanner Process) Non-RPF Multicast Traffic show Commands Exec Processes L3 Aging Process BPDU Storm SPAN Sessions %CFIB-SP-STBY-7-CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched Copper SPFs Modular IOS Check CPU Utilization Utilities and Tools to Determine the Traffic That Is Punted to the CPU Cisco IOS System Software CatOS System Software Recommendations NetPro Discussion Forums - Featured Conversations Related Information Introduction This document describes causes of high CPU utilization on Cisco Catalyst 6500/6000 Series Switches and Virtual Switching System (VSS) 1440 based systems. Like Cisco routers, switches use the show processes cpu command in order to show CPU utilization for the switch supervisor engine processor. However, due to
Transcript
Catalyst 6500/6000 Switch High CPU Utilization
Document ID: 63992
IntroductionPrerequisites Requirements Components Used ConventionsDifference Between CatOS and Cisco IOS System SoftwareUnderstand CPU Utilization on Catalyst 6500/6000 SwitchesSituations and Features That Trigger Traffic to Go to Software Packets That Are Destined to the Switch Packets and Conditions That Require Special Processing ACL−Based Features NetFlow−Based Features Multicast Traffic Other Features IPv6 Situations LCP Schedular and DFC ModuleCommon Causes and Solutions for High CPU Utilization Issues IP Unreachables NAT Translations Use of CEF FIB Table Space in the Flow Cache Table Optimized ACL Logging Rate Limit of Packets to the CPU Physical Merger of VLANs Due to Incorrect Cabling Broadcast Storm BGP Next−Hop Address Tracking (BGP Scanner Process) Non−RPF Multicast Traffic show Commands Exec Processes L3 Aging Process BPDU Storm SPAN Sessions %CFIB−SP−STBY−7−CFIB_EXCEPTION : FIB TCAM exception, Some entries will besoftware switched Copper SPFs Modular IOSCheck CPU UtilizationUtilities and Tools to Determine the Traffic That Is Punted to the CPU Cisco IOS System Software CatOS System SoftwareRecommendationsNetPro Discussion Forums − Featured ConversationsRelated Information
Introduction
This document describes causes of high CPU utilization on Cisco Catalyst 6500/6000 Series Switches andVirtual Switching System (VSS) 1440 based systems. Like Cisco routers, switches use the show processescpu command in order to show CPU utilization for the switch supervisor engine processor. However, due to
the differences in architecture and forwarding mechanisms between Cisco routers and switches, the typicaloutput of the show processes cpu command differs significantly. The meaning of the output differs, too. Thisdocument clarifies these differences. The document describes use of the CPU on the switches and how tointerpret the show processes cpu command output.
Note: In this document, the words "switch" and "switches" refer to the Catalyst 6500/6000 Switches.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
The information in this document is based on the software and hardware versions for Catalyst 6500/6000Switches and Virtual Switching System (VSS) 1440 based systems.
The information in this document was created from the devices in a specific lab environment. All of thedevices used in this document started with a cleared (default) configuration. If your network is live, make surethat you understand the potential impact of any command.
Note: The supported software for Virtual Switching System (VSS) 1440 based systems is Cisco IOS®
Software Release 12.2(33)SXH1 or later.
Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.
Difference Between CatOS and Cisco IOS System Software
Catalyst OS (CatOS) on the Supervisor Engine and Cisco IOS® Software on the Multilayer SwitchFeature Card (MSFC) (Hybrid): You can use a CatOS image as the system software to run the supervisorengine on Catalyst 6500/6000 Switches. If the optional MSFC is installed, a separate Cisco IOS Softwareimage is used to run the MSFC.
Cisco IOS Software on both the Supervisor Engine and MSFC (Native): You can use a single Cisco IOSSoftware image as the system software to run both the supervisor engine and MSFC on Catalyst 6500/6000Switches.
Note: Refer to Comparison of the Cisco Catalyst and Cisco IOS Operating Systems for the Cisco Catalyst6500 Series Switch for more information.
Understand CPU Utilization on Catalyst 6500/6000 Switches
Cisco software−based routers use software in order to process and route packets. CPU utilization on a Ciscorouter tends to increase as the router performs more packet processing and routing. Therefore, the showprocesses cpu command can provide a fairly accurate indication of the traffic processing load on the router.
Catalyst 6500/6000 Switches do not use the CPU in the same way. These switches make forwarding decisionsin hardware, not in software. Therefore, when the switches make the forwarding or switching decision formost frames that pass through the switch, the process does not involve the supervisor engine CPU.
On Catalyst 6500/6000 Switches, there are two CPUs. One CPU is the supervisor engine CPU, which is calledthe Network Management Processor (NMP) or the Switch Processor (SP). The other CPU is the Layer 3routing engine CPU, which is called the MSFC or the Route Processor (RP).
The SP CPU performs functions that include:
Assists in MAC address learning and aging
Note: MAC address learning is also called path setup.
•
Runs protocols and processes that provide network control
Examples include Spanning Tree Protocol (STP), Cisco Discovery Protocol (CDP), VLAN TrunkProtocol (VTP), Dynamic Trunking Protocol (DTP), and Port Aggregation Protocol (PAgP).
•
Handles network management traffic that is destined to the CPU of the switch
Examples include Telnet, HTTP, and Simple Network Management Protocol (SNMP) traffic.
•
The RP CPU performs functions that include:
Builds and updates the Layer 3 routing and Address Resolution Protocol (ARP) tables• Generates the Cisco Express Forwarding (CEF) Forwarding Information Base (FIB) and adjacencytables, and downloads the tables into the Policy Feature Card (PFC)
•
Handles network management traffic that is destined to the RP
Examples include Telnet, HTTP, and SNMP traffic.
•
Situations and Features That Trigger Traffic to Go toSoftware
Packets That Are Destined to the Switch
Any packet that is destined to the switch goes to software. Such packets include:
Control packets
Control packets are received for STP, CDP, VTP, Hot Standby Router Protocol (HSRP), PAgP, LinkAggregation Control Protocol (LACP), and UniDirectional Link Detection (UDLD).
•
Routing protocol updates
Examples of these protocols are Routing Information Protocol (RIP), Enhanced Interior GatewayRouting Protocol (EIGRP), Border Gateway Protocol (BGP), and Open Shortest Path First Protocol(OSPF Protocol).
•
SNMP traffic that is destined to the switch• Telnet and Secure Shell Protocol (SSH) traffic to the switch• ARP responses to ARP requests•
Packets and Conditions That Require Special Processing
This list provides specific packet types and conditions that force packets to be handled in software:
Packets with IP options, an expired Time to Live (TTL), or non−Advanced Research Projects Agency(ARPA) encapsulation
•
Packets with special handling, such as tunneling• IP fragmentation• Packets that require Internet Control Message Protocol (ICMP) messages from the RP or SP• Maximum transmission unit (MTU) check failure• Packets with IP errors, which include IP checksum and length errors• Adjacency same interface• Packets that fail the Reverse Path Forwarding (RPF) check�rpf−failure• Glean/receive
Glean refers to packets that require ARP resolution, and receive refers to packets that fall in thereceive case.
•
Internetwork Packet Exchange (IPX) traffic that is software−switched on the Supervisor Engine 720in both Cisco IOS Software and CatOS
IPX traffic is also software−switched on the Supervisor Engine 2/Cisco IOS Software, but the trafficis hardware−switched on the Supervisor Engine 2/CatOS. IPX traffic is hardware−switched on theSupervisor Engine 1A for both operating systems.
•
AppleTalk traffic• Hardware resources full conditions
These resources include FIB, content−addressable memory (CAM), and ternary CAM (TCAM).
•
ACL−Based Features
Access control list (ACL)−denied traffic with the ICMP unreachables feature turned on
Note: This is the default.
Some ACL−denied packets are leaked to the MSFC if IP unreachables are enabled. Packets thatrequire ICMP unreachables are leaked at a user−configurable rate. By default, the rate is 500 packetsper second (pps).
•
IPX filtering on the basis of unsupported parameters, such as source host
On the Supervisor Engine 720, the process of Layer 3 IPX traffic is always in software.
•
Access control entries (ACEs) that require logging, with the log keyword
This applies to ACL log and VLAN ACL (VACL) log features. ACEs in the same ACL that do notrequire logging still process in hardware. The Supervisor Engine 720 with PFC3 supports the ratelimit of packets that are redirected to the MSFC for ACL and VACL logging. The Supervisor Engine2 supports the rate limit of packets that are redirected to the MSFC for VACL logging. Support forACL logging on the Supervisor Engine 2 is scheduled for the Cisco IOS Software Release 12.2Sbranch.
•
Policy−routed traffic, with use of match length, set ip precedence, or other unsupported parameters
The set interface parameter has support in software. However, the set interface null 0 parameter isan exception. This traffic is handled in hardware on the Supervisor Engine 2 with PFC2 and theSupervisor Engine 720 with PFC3.
•
Non−IP and non−IPX router ACLs (RACLs)
Non−IP RACLs apply to all supervisor engines. The non−IPX RACLs apply to the Supervisor Engine1a with PFC and the Supervisor Engine 2 with PFC2 only.
•
Broadcast traffic that is denied in an RACL• Traffic that is denied in a unicast RPF (uRPF) check, ACL ACE•
This uRPF check applies to the Supervisor Engine 2 with PFC2 and Supervisor Engine 720 withPFC3.Authentication proxy
Traffic that is subject to authentication proxy can be rate−limited on the Supervisor Engine 720.
•
Cisco IOS Software IP Security (IPsec)
Traffic that is subject to Cisco IOS encryption can be rate−limited on the Supervisor Engine 720.
•
NetFlow−Based Features
The NetFlow−based features that this section describes apply to the Supervisor Engine 2 and SupervisorEngine 720 only.
NetFlow−based features always need to see the first packet of a flow in software. Once the firstpacket of the flow reaches software, subsequent packets for the same flow are hardware−switched.
This flow arrangement applies to reflexive ACLs, Web Cache Communication Protocol (WCCP), andCisco IOS Server Load Balancing (SLB).
Note: On the Supervisor Engine 1, reflexive ACLs rely on dynamic TCAM entries to create hardwareshortcuts for a particular flow. The principle is the same: the first packet of a flow goes to software.Subsequent packets for that flow are hardware−switched.
•
With the TCP Intercept feature, the three−way handshake and session close are handled in software.The rest of the traffic is handled in hardware.
Note: Synchronize (SYN), SYN acknowledge (SYN ACK), and ACK packets comprise thethree−way handshake. Session close occurs with finish (FIN) or reset (RST).
•
With Network Address Translation (NAT), traffic is handled in this way:
On the Supervisor Engine 720:
Traffic that requires NAT is handled in hardware after the initial translation. Translation ofthe first packet of a flow occurs in software, and subsequent packets for that flow arehardware−switched. For TCP packets, a hardware shortcut is created in the NetFlow tableafter completion of the TCP three−way handshake.
♦
On the Supervisor Engine 2 and Supervisor Engine 1:
All traffic that requires NAT is software−switched.
♦
•
Context−based Access Control (CBAC) uses NetFlow shortcuts in order to classify traffic thatrequires inspection. Then, CBAC sends only this traffic to software. CBAC is a software−onlyfeature; traffic that is subject to inspection is not hardware−switched.
Note: Traffic that is subject to inspection can be rate−limited on the Supervisor Engine 720.
•
Multicast Traffic
Protocol Independent Multicast (PIM) snooping• Internet Group Management Protocol (IGMP) snooping (TTL = 1)
This traffic is indeed destined to the router.FIB miss• Multicast packets for registration that have direct connection to the multicast source
These multicast packets are tunneled to the rendezvous point.
•
IP version 6 (IPv6) multicast•
Other Features
Network−Based Application Recognition (NBAR)• ARP Inspection, with CatOS only• Port Security, with CatOS only• DHCP snooping•
IPv6 Situations
Packets with a hop−by−hop option header• Packets with the same destination IPv6 address as that of routers• Packets that fail the scope enforcement check• Packets that exceed the MTU of the output link• Packets with a TTL that is less than or equal to 1• Packets with an input VLAN that equals the output VLAN• IPv6 uRPF
Software handles this tunneling. All other traffic that enters an ISATAP tunnel is hardware−switched.
•
LCP Schedular and DFC Module
In a Distributed Forwarding Card (DFC), the lcp schedular process that runs on a high CPU is not anissue and does not pose any problem to the operation. The LCP schedular is part of the firmware code. On allmodules that do not require a DFC, the firmware runs on a specific processor called the Line Card Processor(LCP). This processor is used to program the ASIC hardware and to communicate to the central supervisormodule.
When the lcp schedular is initiated, it makes use of all the process time available. But when a newprocess needs processor time, lcp schedular frees up process time for the new process. There is noimpact to the performance of the system with regard to this high CPU utilization. The process simply grabs allunused CPU cycles, as long as no higher priority process requires them.
Common Causes and Solutions for High CPU UtilizationIssues
IP Unreachables
When an access group denies a packet, the MSFC sends ICMP unreachable messages. This action occurs bydefault.
With the default enablement of the ip unreachables command, the supervisor engine drops most of thedenied packets in hardware. Then, the supervisor engine sends only a small number of packets, a maximum of10 pps, to the MSFC for drop. This action generates ICMP−unreachable messages.
The drop of denied packets and generation of ICMP−unreachable messages imposes a load on the MSFCCPU. In order to eliminate the load, you can issue the no ip unreachables interface configuration command.This command disables ICMP−unreachable messages, which allows the drop in hardware of all accessgroup−denied packets.
ICMP−unreachable messages are not sent if a VACL denies a packet.
NAT Translations
NAT utilizes both hardware and software forwarding. The initial establishment of the NAT tranlsations mustbe done in software and further forwarding is done with hardware. NAT also utilizes the Netflow table (128KB maximum). Therefore, if the Netflow table is full, the switch will also start to apply NAT forwarding viasoftware. This normally happens with high traffic bursts and will cause an increase on the CPU of 6500.
Use of CEF FIB Table Space in the Flow Cache Table
The Supervisor Engine 1 has a Flow Cache Table that supports 128,000 entries. However, on the basis of theefficiency of the hashing algorithm, these entries range from 32,000 to 120,000. On the Supervisor Engine 2,the FIB table is generated and programmed into the PFC. The table holds as many as 256,000 entries. TheSupervisor Engine 720 with PFC3−BXL supports up to 1,000,000 entries. Once this space is exceeded, thepackets become switched in software. This can cause high CPU utilization on the RP. In order to check thenumber of routes in the CEF FIB table, use these commands:
Router#show processes cpuCPU utilization for five seconds: 99.26% one minute: 100.00% five minutes: 100.00%
IOS% show ip cef summaryIP Distributed CEF with switching (Table Version 86771423), flags=0x0 112564 routes, 1 reresolve, 0 unresolved (0 old, 0 new) 112567 leaves, 6888 nodes, 21156688 bytes, 86771426inserts, 86658859invalidations 295 load sharing elements, 96760 bytes, 112359 references universal per−destination load sharing algorithm, id 8ADDA64A 2 CEF resets, 2306608 revisions of existing leaves refcounts: 1981829 leaf, 1763584 node
!−−− You see these messages if the TCAM space is exceeded:
%MLSCEF−SP−7−FIB_EXCEPTION: FIB TCAM exception, Some entries will be software switched%MLSCEF−SP−7−END_FIB_EXCEPTION: FIB TCAM exception cleared, all CEF entries will be hardware switched
On the Supervisor Engine 2, the number of FIB entries reduces to half if you have configured RPF check onthe interfaces. This configuration can lead to the software switch of more packets and, consequently, highCPU utilization.
Refer to Understanding ACL on Catalyst 6500 Series Switches for additional information about TCAMutilization and optimization.
Optimized ACL Logging
Optimized ACL Logging (OAL) provides hardware support for ACL logging. Unless you configure OAL, theprocess of packets that require logging takes place completely in software on the MSFC3. OAL permits ordrops packets in hardware on the PFC3. OAL uses an optimized routine to send information to the MSFC3 inorder to generate the logging messages.
Note: For information about OAL, refer to the Optimized ACL Logging with a PFC3 section ofUnderstanding Cisco IOS ACL Support.
Rate Limit of Packets to the CPU
On the Supervisor Engine 720, rate limiters can control the rate at which packets can go to software. This ratecontrol helps prevent denial−of−service attacks. You can also use a few of these rate limiters on theSupervisor Engine 2:
Router#show mls rate−limit Rate Limiter Type Status Packets/s Burst−−−−−−−−−−−−−−−−−−−−− −−−−−−−−−− −−−−−−−−− −−−−− MCAST NON RPF Off − − MCAST DFLT ADJ On 100000 100 MCAST DIRECT CON Off − − ACL BRIDGED IN Off − − ACL BRIDGED OUT Off − − IP FEATURES Off − − ACL VACL LOG On 2000 1 CEF RECEIVE Off − − CEF GLEAN Off − − MCAST PARTIAL SC On 100000 100 IP RPF FAILURE On 500 10 TTL FAILURE Off − −ICMP UNREAC. NO−ROUTE On 500 10ICMP UNREAC. ACL−DROP On 500 10 ICMP REDIRECT Off − − MTU FAILURE Off − − LAYER_2 PDU Off − − LAYER_2 PT Off − − IP ERRORS On 500 10 CAPTURE PKT Off − − MCAST IGMP Off − −
Router(config)#mls rate−limit ? all Rate Limiting for both Unicast and Multicast packets layer2 layer2 protocol cases multicast Rate limiting for Multicast packets unicast Rate limiting for Unicast packets
Here is an example:
Router(config)#mls rate−limit layer2 l2pt 3000
In order to rate−limit all CEF−punted packets to the MSFC, issue the command that is in this example:
Router(config)#mls ip cef rate−limit 50000
In order to reduce the number of packets punted to the CPU due to TTL=1, issue this command:
Router(config)#mls rate−limit all ttl−failure 15
!−−− where 15 is the number of packets per second with TTL=1.!−−− The valid range is from 10 to 1000000 pps.
High CPU can also be due to packets with TTL=1 which are leaked to the CPU. In order to limit the numberof packets that are leaked to the CPU, configure a hardware rate limiter. Rate limiters can rate−limit packetsthat are leaked from the hardware data path up to the software data path. Rate limiters protect the softwarecontrol path from congestion by dropping the traffic that exceeds the configured rate. The rate limit isconfigured using the mls rate−limit all ttl−failure command.
Physical Merger of VLANs Due to Incorrect Cabling
High CPU utilization also can result from the merge together of two or more VLANs due to improper cabling.Also, if STP is disabled on those ports where the VLAN merger happens, high CPU utilization can occur.
In order to resolve this problem, identify the cabling errors and correct them. If your requirement allows, youcan also enable STP on those ports.
Broadcast Storm
A LAN broadcast storm occurs when broadcast or multicast packets flood the LAN, which creates excessivetraffic and degrades network performance. Errors in the protocol−stack implementation or in the networkconfiguration can cause a broadcast storm.
Due to the architectural design of the Catalyst 6500 series platform, the broadcast packets are only and alwaysdropped at the software level.
Broadcast suppression prevents the disruption of LAN interfaces by a broadcast storm. Broadcast suppressionuses filtering that measures broadcast activity on a LAN over a 1−second time period and compares themeasurement with a predefined threshold. If the threshold is reached, further broadcast activity is suppressedfor the duration of a specified time period. Broadcast suppression is disabled by default.
In order to understand how broadcast suppression works and to enable the feature, refer to:
Configuring Broadcast Suppression (Cisco IOS system software)• Configuring Broadcast Suppression (CatOS system software)•
The BGP Scanner process walks the BGP table and confirms reachability of the next hops. This process alsochecks conditional advertisement in order to determine whether BGP should advertise condition prefixesand/or perform route dampening. By default, the process scans every 60 seconds.
You can expect high CPU utilization for short durations because of the BGP Scanner process on a router thatcarries a large Internet routing table. Once per minute, the BGP Scanner walks the BGP Routing InformationBase (RIB) table and performs important maintenance tasks. These tasks include:
A check of the next hop that is referenced in the router BGP table• Verification that the next−hop devices can be reached•
Thus, a large BGP table takes an equivalently large amount of time to be walked and validated. The BGPScanner process walks the BGP table in order to update any data structures and walks the routing table forroute redistribution purposes. Both tables are stored separately in the router memory. Both tables can be verylarge and, thus, consume CPU cycles.
For more information on CPU utilization by the BGP Scanner process, refer to the High CPU due to BGPScanner section of Troubleshooting High CPU Caused by the BGP Scanner or BGP Router Process.
For more information on the BGP Next−Hop Address Tracking feature and the procedure to enable/disable oradjust the scan interval, refer to BGP Support for Next−Hop Address Tracking.
Non−RPF Multicast Traffic
Multicast routing (unlike unicast routing) is only concerned with the source of a given multicast data stream.That is, the IP address of the device that originates the multicast traffic. The basic principle is that the sourcedevice "pushes" the stream out to an undefined number of receivers (within its multicast group). All multicastrouters create distribution trees, which control the path that multicast traffic takes through the network inorder to deliver traffic to all receivers. The two basic types of multicast distribution trees are source trees andshared trees. RPF is a key concept in multicast forwarding. It enables routers to correctly forward multicasttraffic down the distribution tree. RPF makes use of the existing unicast routing table to determine theupstream and downstream neighbors. A router forwards a multicast packet only if it is received on theupstream interface. This RPF check helps to guarantee that the distribution tree is loop−free.
Multicast traffic is always visible by every router on a bridged (Layer 2) LAN, according to the IEEE 802.3CSMA/CD specification. In the 802.3 standard, bit 0 of the first octet is used to indicate a broadcast and/ormulticast frame, and any Layer 2 frame with this address is flooded. This is also the case even if CGMP orIGMP Snooping are configured. This is because multicast routers must see the multicast traffic, if they areexpected to make a proper forwarding decision. If multiple multicast routers each have interfaces onto acommon LAN, then only one router forwards the data (chosen by an election process). Due to the floodingnature of LANs, the redundant router (router that does not forward the multicast traffic) receives this data onthe outbound interface for that LAN. The redundant router normally drops this traffic, because it has arrivedon the wrong interface and therefore fails the RPF check. This traffic that fails the RPF check is callednon−RPF traffic or RPF failure packets, because they have been transmitted backwards against the flow fromthe source.
The Catalyst 6500 with an MSFC installed, can be configured to act as full−fledged multicast router. UtilizingMulticast Multi−Layer Switching (MMLS), RPF traffic is usually forwarded by the hardware within theswitch. The ASICs are given information from the multicast routing state ( for example, (*,G) and (S,G) ), sothat a hardware shortcut can be programmed into the Netflow and/or FIB table. This non−RPF traffic is stillnecessary in some cases, and is required by the MSFC CPU (at process level) for the PIM Assert mechanism.Otherwise, it is then dropped by the software fast−switching path (assumption is that software fast−switchingis not disabled on the RPF interface).
The Catalyst 6500 that uses redundancy might not handle non−RPF traffic efficiently in certain topologies.For non−RPF traffic, there is usually no (*,G) or (S,G) state in the redundant router, and therefore nohardware or software shortcuts can be created to drop the packet. Each multicast packet must be examined bythe MSFC route processor individually, and this is often referred to as CPU Interrupt traffic. With Layer 3hardware switching and multiple interfaces/VLANs that connect the same set of routers, the non−RPF trafficthat hits the CPU of the redundant MSFC is amplified "N" times the original source rate (where "N" is thenumber of LANs to which the router is redundantly connected). If the rate of the non−RPF traffic exceeds thepacket dropping capacity of the system, then it might cause high CPU utilization, buffer overflows and overallnetwork instability.
With the Catalyst 6500, there is an access list engine that enables filtering to take place at wire rate. Thisfeature can be used to handle non−RPF traffic for Sparse Mode groups efficiently, in certain situations. Youcan only use the ACL−based method within sparse−mode 'stub networks', where there are no downstreammulticast routers (and corresponding receivers). Additionally, because of the packet forwarding design of theCatalyst 6500, internally redundant MSFCs cannot use this implementation. This is outlined within the Ciscobug ID CSCdr74908 ( registered customers only) . For dense−mode groups, non−RPF packets must be seen on therouter for the PIM Assert mechanism to function properly. Different solutions, such as CEF or Netflow basedrate−limiting and QoS are used to control RPF failures in dense−mode networks and sparse−mode transitnetworks.
On the Catalyst 6500 there is an access list engine that enables filtering to take place at wire rate. This featurecan be used to handle non−RPF traffic for Sparse Mode groups efficiently. In order to implement this
solution, place an access list on the incoming interface of the `stub network' to filter multicast traffic that didnot originate from the `stub network'. The access list is pushed down to the hardware in the switch. Thisaccess list prevents the CPU from ever seeing the packet and allows the hardware to drop the non−RPF traffic.
Note: Do not place this access list on a transit interface. It is only intended for stub networks (networks withhosts only).
Refer to these documents for more information:
Redundant Router Issues with IP Multicast in Stub Networks• Non−RPF Traffic Processing•
show Commands
The CPU utilization when you issue a show command is always almost 100%. It is normal to have high CPUutilization when you issue a show command and normally remains for only a few seconds.
For example, it is normal for the Virtual Exec process to go high when you issue a show tech−supportcommand as this output is an interrupt driven output. Your only concern having high CPU in other processesother than show commands.
Exec Processes
The Exec process in Cisco IOS Software is responsible for communication on the TTY lines (console,auxiliary, asynchronous) of the router. The Virtual Exec process is responsible for the VTY lines (Telnetsessions). The Exec and Virtual Exec processes are medium priority processes, so if there are other processesthat have a higher priority (High or Critical), the higher priority processes get the CPU resources.
If there is a lot of data transferred through these sessions, the CPU utilization for the Exec process increases.This is because when the router wants to send a simple character through these lines, the router uses someCPU resources:
For the console (Exec), the router uses one interrupt per character.• For the VTY line (Virtual Exec), the Telnet session has to build one TCP packet per character.•
This list details some of the possible reasons for high CPU utilization in the Exec process:
There is too much data sent through the console port.
Check to see if any debugs have been started on the router with the show debuggingcommand.
1.
Disable console logging on the router with the no form of the logging console command.2. Verify if a long output is printed on the console. For example, a show tech−support or ashow memory command.
3.
•
The exec command is configured for asynchronous and auxiliary lines.
If a line has only outgoing traffic, disable the Exec process for this line. This is because if thedevice (for example, a modem) attached to this line sends some unsolicited data, the Execprocess starts on this line.
1.
If the router is used as a terminal server (for reverse Telnet to other device consoles), it isrecommended that you configure the no exec command on the lines that are connected to theconsole of the other devices. Data that comes back from the console might otherwise start anExec process, which uses CPU resources.
2.
•
A possible reason for high CPU utilization in the Virtual Exec process is:
There is too much data sent across the Telnet sessions.
The most common reason for high CPU utilization in the Virtual Exec process is that too much data istransferred from the router to the Telnet session. This can happen when commands with long outputssuch as show tech−support, show memory, and so on, are executed from the Telnet session. Theamount of data transferred through each VTY session can be verified with the show tcp vty <linenumber> command.
•
L3 Aging Process
When the L3 aging process exports a large number of ifindex values using NetFlow Data Export (NDE), theCPU usage might hit 100%.
If you encounter this problem, check whether these two commands are enabled:
set mls nde destination−ifindex enable
set mls nde source−ifindex enable
If you enable these commands, the process must export all destination and source ifindex values using NDE.The L3 aging process utilization goes high since it must perform FIB lookup for all destination and sourceifindex values. Because of this, the table becomes full, the L3 aging process goes high, and the CPU usage hits100%.
In order to resolve this issue, disable these commands:
set mls nde destination−ifindex disable
set mls nde source−ifindex disable
BPDU Storm
Spanning tree maintains a loop−free Layer 2 environment in redundant switched and bridges networks.Without STP, frames loop and/or multiply indefinitely. This occurrence causes a network meltdown becausehigh traffic interrupts all devices in the broadcast domain.
In some respects, STP is an early protocol that was initially developed for slow software−based bridgespecifications (IEEE 802.1D), but STP can be complicated in order to implement it successfully in largeswitched networks that have these features:
Many VLANs• Many switches in a STP domain• Multi−vendor support• Newer IEEE enhancements•
If the network faces frequent spanning tree calculations or the switch has to process more BPDUs, it can resultin high CPU, as well as BPDU drops.
In order to work around these issues, perform any or all of these steps:
Prune off the VLANs from the switches.1. Use an enhanced version of STP, such as MST.2.
Upgrade the hardware of the switch.3.
Also refer to best practices to implement Spanning Tree Protocol in the network.
Best Practices for Catalyst 4500/4000, 5500/5000, and 6500/6000 Series Switches Running CatOSConfiguration and Management
•
Best Practices for Catalyst 6500/6000 Series and Catalyst 4500/4000 Series Switches Running CiscoIOS Software
•
SPAN Sessions
Based on the architecture of Catalyst 6000/6500 Series Switches, SPAN sessions do not affect theperformance of the switch, but, if the SPAN session includes a high traffic / uplink port or an EtherChannel, itcan increase the load on the processor. If it then singles out a specific VLAN, it increases the workload evenmore. If there is bad traffic on the link, that can further increase the workload.
In some scenarios, the RSPAN feature can cause loops, and the load on the processor shoots up. For moreinformation, refer to Why Does the SPAN Session Create a Bridging Loop?
The switch can pass traffic as usual since everything is in hardware, but the CPU can take a beating if it triesto figure out which traffic to send through. It is recommended that you configure SPAN sessions only when itis required.
%CFIB−SP−STBY−7−CFIB_EXCEPTION : FIB TCAM exception, Someentries will be software switched
%CFIB−SP−7−CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched %CFIB−SP−STBY−7−CFIB_EXCEPTION : FIB TCAM exception, Some entries will be softwareswitched
This error message is received when the amount of available space in the TCAM is exceeded. This results inhigh CPU. This is a FIB TCAM limitation. Once TCAM is full, a flag will be set and FIB TCAM exception isreceived. This stops from adding new routes to the TCAM. Therefore, everything will be software switched.The removal of routes does not help resume hardware switching. Once the TCAM enters the exception state,the system must be reloaded to get out of that state. The maximum routes that can be installed in TCAM isincreased by the mls cef maximum−routes command.
Copper SPFs
In Cisco ME 6500 series Ethernet switches, the copper SFPs require more firmware interaction than othertypes of SFPs, which increases the CPU utilization.
The software algorithms that manage copper SFPs have been improved in the Cisco IOS SXH releases.
Modular IOS
In Cisco Catalyst 6500 series switches that run modular IOS software, the normal CPU utilization is a littlegreater than non−modular IOS software.
Modular IOS software pays a price per activity more than it pays a price per packet. Modular IOS softwaremaintains the processes by consuming certain CPU even if there is no much packets, so the CPU consumptionis not based on the actual traffic. However, when packets been processed go high rate, the CPU consumed inModular IOS software should not be more than that in non−modular IOS software.
Check CPU Utilization
If the CPU utilization is high, issue the show processes cpu command first. The output shows you the CPUutilization on the switch as well as the CPU consumption by each process.
Router#show processes cpuCPU utilization for five seconds: 57%/48%; one minute: 56%; five minutes: 48% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 0 5 0 0.00% 0.00% 0.00% 0 Chunk Manager 2 12 18062 0 0.00% 0.00% 0.00% 0 Load Meter 4 164532 13717 11994 0.00% 0.21% 0.17% 0 Check heaps 5 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager
In this output, the total CPU utilization is 57 percent and the interrupt CPU utilization is 48 percent. Here,these percentages appear in boldface text. The interrupt switch of traffic by the CPU causes the interrupt CPUutilization. The command output lists the processes that cause the difference between the two utilizations. Inthis case, the cause is the SNMP process.
On the supervisor engine that runs CatOS, the output looks like this:
Switch> (enable) show processes cpu
CPU utilization for five seconds: 99.72% one minute: 100.00% five minutes: 100.00%
In this output, the first process is Kernel and Idle, which shows idle CPU utilization. This process isnormally high, unless some other processes consume CPU cycles. In this example, the SptBpduRx processcauses high CPU utilization.
If the CPU utilization is high due to one of these processes, you can troubleshoot and determine why thisprocess runs high. But, if the CPU is high due to traffic being punted to the CPU, you need to determine whythe traffic is being punted. This determination can help you identify what the traffic is.
Utilities and Tools to Determine the Traffic That Is Punted tothe CPU
This section identifies some utilities and tools that can help you look at this traffic.
Cisco IOS System Software
In Cisco IOS Software, the switch processor on the supervisor engine is referred to as the SP, and the MSFCis called the RP.
The show interface command gives basic information on the state of the interface and the traffic rate on theinterface. The command also provides error counters.
Router#show interface gigabitethernet 4/1GigabitEthernet4/1 is up, line protocol is up (connected) Hardware is C6k 1000Mb 802.3, address is 000a.42d1.7580 (bia 000a.42d1.7580) Internet address is 100.100.100.2/24 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half−duplex, 100Mb/s input flow−control is off, output flow−control is off Clock mode is auto ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 5/75/1/24075 (size/max/drops/flushes); Total output drops: 2 Queueing strategy: fifo Output queue: 0/40 (size/max) 30 second input rate 7609000 bits/sec, 14859 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec
In this output, you can see that the incoming traffic is Layer 3−switched instead of Layer 2−switched. Thisindicates that the traffic is being punted to the CPU.
The show processes cpu command tells you whether these packets are regular traffic packets or controlpackets.
Router#show processes cpu | exclude 0.00CPU utilization for five seconds: 91%/50%; one minute: 89%; five minutes: 47% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 5 881160 79142 11133 0.49% 0.19% 0.16% 0 Check heaps 98 121064 3020704 40 40.53% 38.67% 20.59% 0 IP Input 245 209336 894828 233 0.08% 0.05% 0.02% 0 IFCOM Msg Hdlr
If the packets are process−switched, you see that the IP Input process runs high. Issue this command in
If the traffic is interrupt switched, you cannot see those packets with the show buffers input−interfacecommand. In order to see the packets that are punted to the RP for interrupt switching, you can perform aSwitched Port Analyzer (SPAN) capture of the RP port.
Note: Refer to this document for additional information about interrupt−switched versus process−switchedCPU utilization:
High CPU Utilization due to Interrupts section of Troubleshooting High CPU Utilization on CiscoRouters
•
SPAN RP−Inband and SP−Inband
A SPAN for the RP or SP port in Cisco IOS Software is available in Cisco IOS Software Release 12.1(19)Eand later.
Note: For the SXH release, you must use the monitor session command in order to configure a local SPANsession, and then use this command to associate the SPAN session with the CPU:
Note: For more information on these commands, refer to Configuring Local SPAN (SPAN ConfigurationMode) in the Catalyst 6500 Release 12.2SX Software Configuration Guide.
Here is an example on an RP console:
Router#monitor session 1 source interface fast 3/3
!−−− Use any interface that is administratively shut down.
For switches that run CatOS system software, the supervisor engine runs CatOS and the MSFC runs CiscoIOS Software.
If you issue the show mac command, you can see the number of frames that are punted to the MSFC. Port15/1 is the supervisor engine connection to the MSFC.
Note: The port is 16/1 for supervisor engines in slot 2.
Console> (enable) show mac 15/1
Port Rcv−Unicast Rcv−Multicast Rcv−Broadcast−−−−−−−− −−−−−−−−−−−−−−−−−−−− −−−−−−−−−−−−−−−−−−−− −−−−−−−−−−−−−−−−−−−−15/1 193576 0 1
Port Xmit−Unicast Xmit−Multicast Xmit−Broadcast−−−−−−−− −−−−−−−−−−−−−−−−−−−− −−−−−−−−−−−−−−−−−−−− −−−−−−−−−−−−−−−−−−−−15/1 3 0 0
A quick increase in this number indicates that packets are punted to the MSFC, which causes high CPUutilization. You can then look at the packets in these ways:
SPAN MSFC port 15/1 or 16/1• SPAN sc0•
SPAN MSFC Port 15/1 or 16/1
Set up a SPAN session in which the source is the MSFC port 15/1 (or 16/1) and the destination is an Ethernetport.
Here is an example:
Console> (enable) set span 15/1 5/10Console> (enable) show span
Destination : Port 5/10Admin Source : Port 15/1Oper Source : NoneDirection : transmit/receiveIncoming Packets: disabledLearning : enabledMulticast : enabledFilter : −Status : active
If you collect a sniffer trace on port 5/10, the sniffer trace shows packets that transmit to and from theMSFC. Configure the SPAN session as tx in order to capture packets that are only destined to the MSFC, andnot from the MSFC.
SPAN sc0
Set up a SPAN session with the sc0 interface as the source in order to capture frames that go to the supervisorengine CPU.
Console> (enable) set span ? disable Disable port monitoring
sc0 Set span on interface sc0 <mod/port> Source module and port numbers <vlan> Source VLAN numbers
Note: For Optical Services Modules (OSMs), you cannot perform a SPAN capture of traffic.
Recommendations
The supervisor engine CPU utilization does not reflect the hardware forwarding performance of the switch.Still, you must baseline and monitor the supervisor engine CPU utilization.
Baseline the supervisor engine CPU utilization for the switch in a steady−state network with normaltraffic patterns and load.
1.
Note which processes generate the highest CPU utilization.When you troubleshoot CPU utilization, consider these questions:
Which processes generate the highest utilization? Are these processes different from yourbaseline?
♦
Is the CPU consistently elevated, over the baseline? Or are there spikes of high utilization,and then a return to the baseline levels?
♦
Are there Topology Change Notifications (TCNs) in the network?
Note: Flapping ports or host ports with STP PortFast disabled cause TCNs.
♦
Is there excessive broadcast or multicast traffic in the management subnets/VLAN?♦ Is there excessive management traffic, such as SNMP polling, on the switch?♦
2.
If possible, isolate the management VLAN from the VLANs with user data traffic, particularly heavybroadcast traffic.
Examples of this type of traffic include IPX RIP/Service Advertising Protocol (SAP), AppleTalk, andother broadcast traffic. Such traffic can impact the supervisor engine CPU utilization and, in extremecases, can interfere with the normal operation of the switch.
3.
If the CPU runs high due to the punt of traffic to the RP, determine what that traffic is and why thetraffic is punted.
In order to make this determination, use the utilities that the Utilities and Tools to Determine theTraffic That Is Punted to the CPU section describes.
4.
NetPro Discussion Forums − Featured Conversations
Networking Professionals Connection is a forum for networking professionals to share questions, suggestions,and information about networking solutions, products, and technologies. The featured links are some of themost recent conversations available in this technology.
NetPro Discussion Forums − Featured Conversations for LAN
Network Infrastructure: LAN Routing and Switching
Network Infrastructure: Getting Started with LANs
Related Information
Common CatOS Error Messages on Catalyst 6000/6500 Series Switches• Common Error Messages on Catalyst 6500/6000 Series Switches Running Cisco IOS Software• Troubleshooting Hardware and Common Issues on Catalyst 6500/6000 Series Switches RunningCisco IOS System Software
•
Unicast Flooding in Switched Campus Networks• Cisco Catalyst 6500 Series Switches Product Support• LAN Product Support• LAN Switching Technology Support• Technical Support & Documentation − Cisco Systems•
