Troubleshooting Cisco Catalyst 3850 Series Switches BRKCRS-3146
Naoshad Mehta
Principal Engineer
Twitter: @Naoshad #convergedaccess #cat3850 #CLUS
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Troubleshooting Cisco Catalyst 3850 Series Switches Session Overview and Objectives
3
Cisco is bringing together the best of wired and wireless networking
into “One Network” with Converged Access on the Catalyst 3850 Switch
In this session, learn about the capabilities of the 3850 switch and
troubleshoot common issues seen on the 3850 running the IOS-XE
Operating System. Learn about the switch architecture and troubleshooting
hardware, RTU Licensing, Boot-up Sequence, Memory and CPU utilization,
Stacking, High Availability, Forwarding features on the UADP ASIC, QoS and
Wireless.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Your Instructor today … Naoshad Mehta
Principal Engineer, Enterprise Switching Software
4
I’m a Principal Engineer with the Enterprise Switching Software team at Cisco. My current
focus is the adoption of Catalyst 3850 and Converged Access Architecture in the marketplace.
I’ve been with Cisco for 13+ years. My primary responsibility since 2010 was the delivery of
the Catalyst 3850 and CT5760 Wireless Controller. I have been intimately involved with the
design and implementation of almost every software aspect of the 3850 and I’m here to help
you learn more about the architecture and how to troubleshoot the 3850.
Prior to working on the 3850, I have worked on a wide spectrum of technologies (MPLS, Traffic
Engineering, L2VPN, EVCs, etc.), Products (Nexus 7K, 7600, 7500, 7200) and Operating
Systems (Classic IOS, NXOS and IOS-XE).
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Agenda Architecture & Product Overview
Multicast Forwarding
Hardware Troubleshooting
Installer and Bootup
Licensing
Memory and CPU
Unicast Forwarding
CPU Punt Path
Stacking
High Availability
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Glossary
6
A S Active Switch Standby Switch
3850 Switch
MC MA Wireless Mobility Agent
Wireless Mobility Controller
FED – Forwarding Engine Driver WCM – Wireless Controller Module
PDS – Packet Delivery Service UADP – Unified Access Data Plane ASIC
Reference slide that may not be presented in the session
Confusing command/output. Engineering team is committed to fix these.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Suggested Sessions and Reference Material
BRKCRS-2889 - Converged Access System Architecture - Diving into the 'One Network’
BRKCRS-2888 - Converged Access Design Options
BRKARC-3438 - Cisco Catalyst 3850 Series Switching Architecture
Converged Access White Paper – Wired/Wireless System Architecture, Design and Operation - bit.ly/1a4YRh2
7
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Hardware Troubleshooting
Agenda Architecture & Product Overview
Multicast Forwarding
Installer and Bootup
Licensing
Memory and CPU
Unicast Forwarding
CPU Punt Path
Stacking
High Availability
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
9
Overview of the 3850
IOS-XE Evolution
IOS-XE Architecture
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Catalyst 3850 Switch
Bui l t on C isco ’s Innova t i ve “UADP” ASIC
* Roadmap
Wireless CAPWAP Termination
Integrated Controller: Up to 50 APs and 40G per switch
Up to 2000 Clients per Stack
40 Gbps Uplink Bandwidth
Stackpower
Line Rate on All Ports
SGT/SGACL*
480 Gbps Stacking Bandwidth
Full POE+
FRU Fans, Power Supplies
Granular QoS/Flexible
NetFlow
Best-in-Class Wired Switch – with Integrated
Wireless Mobility functionality
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
IOS
IOS 12.2(52)SE
IOS XE Evolution
11
IOS XE 3.2.2(SE)
Management Interface
Module Drivers
Common Infrastructure / HA
I OS -X E
• Modern IOS to enable multi-core CPU
• Easy customer migration
• While maintaining IOS functionality and look and feel
• Allow hosted applications like Wireshark
Management Interface
Module Drivers
Linux Kernel
Common Infrastructure / HA
IOSd
Features Components
Hosted Apps
Features Components WCM
Kernel
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Internal IPC
Avail
ab
ilit
y F
ram
ew
ork
Packet Delivery Service
Service
Location
Forwarding &
Feature Mgr (FFM)
System
Manager
Platform
Manager
Consolidated
Logging
Comet
Services
Licensing
Services
Interface
Manager
Libraries/
Utilities
Services
External
Transports
(TCP/SCTP/UDP)
Wireless Controller HA
Stack Manager (3K)
IOS XE Software Internals Overview
Kernel
IOS
d R
P/L
C
Features PD
Platform
Drivers
Low Level APIs
UADP ASIC
Drivers
Forwarding Engine Driver
12
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Installer and Bootup
Architecture & Product Overview
Agenda
Multicast Forwarding
Hardware Troubleshooting
Licensing
Memory and CPU
Unicast Forwarding
CPU Punt Path
Stacking
High Availability
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
14
Front Panel LEDs
Rear Panel LEDs and Ports
Supported uplink modules
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
System LEDs Overview
15
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
System LEDs – Definitions
• SYST LED
Off = System off
Green = System operating normally
Blinking green = Running POST
Amber = System is malfunctioning
Blinking amber = Network module, power supply, or fan module is malfunctioning
• XPS LED
Off = No XPS cable installed or switch is in StackPower mode
Green = XPS connected and ready to provide backup power
Blinking green = XPS is connected but cannot provide backup power
Amber = XPS is in standby or a fault condition
Blinking amber = Power supply in the switch has failed and is being backed up by XPS
• ACTV LED
Off = Switch is not the active switch
Green = Switch is the active switch or is in standalone mode
Blinking green = Switch is in standby mode
Amber = An error has occurred in the data stack, possibly related to active member selection
• S-PWR LED
Off = StackPower cable not connected or switch is in standalone mode
Green = Switch is connected to an XPS or to 2 StackPower neighbors in a ring configuration
Blinking green = Switch is connected to only 1 StackPower neighbor in a ring configuration
Amber = Fault detected
Blinking amber = StackPower configuration is overbudget
16
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
System LEDs – Definitions (cont.) • STAT LED
Off = Rather than indicating link status, the port LEDs are indicating duplex, speed, stack, or PoE status
Green = Port LEDs are indicating link status
• DUPLX LED
Off = Rather than indicating duplex status, the port LEDs are indicating link, speed, stack, or PoE status
Green = Port LEDs are indicating duplex status
• SPEED LED
Off = Rather than indicating speed status, the port LEDs are indicating link, duplex, stack, or PoE status
Green = Port LEDs are indicating speed status
• STACK LED
Off = Rather than indicating stack status, the port LEDs are indicating link, duplex, speed, or PoE status
Green = Port LEDs are indicating stack status
• PoE LED
Off = Rather than indicating PoE status, the port LEDs are indicating link, duplex, speed, or stack status; None of the downlink ports have been denied power or are in a fault condition
Green = Port LEDs are indicating PoE status and none of the downlink ports have been denied power or are in a fault condition
Blinking amber = Port LEDs are indicating PoE status and at least one of the downlink ports has been denied power or is in a fault condition
• CONSOLE LED
Off = USB console is inactive
Green = USB console is active (RJ45 console is inactive)
17
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Rear Panel Ports & Associated LEDs • CONSOLE SERIAL LED
Off = RJ45 console is inactive (USB console is active)
Green = RJ45 console is active (USB console is inactive)
• MGMT LED
Off = Link down
Green = Link is up with no activity
Blinking green = Link is up with activity
18
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Network Modules
WS-C3850-NM-4-1G
• 4 x 1G
• SFP
• Supported on WS-C3850-24 & WS-C3850-48 Port
WS-C3850-NM-2-10G
• 4 x 1G OR 2 x 10G OR 2 x 1G + 1 x 10G
• SFP & SFP+
• Supported on WS-C3850-24 & WS-C3850-48 Port
WS-C3850-NM-4-10G
• Auto-sensing – All Combinations
• SFP & SFP+
• Supported on WS-C3850-48 only
19
Type of interface 4x1G 2x1G 2x10G 4x10G
1G GigabitEthernet 1 - 4 GigabitEthernet 1 - 2 &
TenGigabitEthernet 3 - 4
TenGigabitEthernet 1 – 4
10G - TenGigabitEthernet 3 - 4
TenGigabitEthernet 1 – 4
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Network Modules – Troubleshooting
• All network modules are hot-pluggable, so they can easily be swapped between units to help narrow down or rule out hardware failures
• As expected, optical modules are also hot-pluggable and can be easily rotated through ports to home in on or rule out hardware failures
If link is down on a module, one often overlooked yet common cause is dirty or damaged fiber
• Each port has a bicolor (green/amber) LED that provides a visual indication of its status
Off = Link down
Solid green = Link up, no activity
Blinking green = Link up with activity
Solid amber = Link disabled
Blinking amber = Link is off due to a fault or because a limit set via software has been exceeded
20
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Licensing
Hardware Troubleshooting
Agenda Architecture & Product Overview
Multicast Forwarding
Installer and Bootup
Memory and CPU
Unicast Forwarding
CPU Punt Path
Stacking
High Availability
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
22
3850 Image naming convention
Packages in the image
Install vs. bundle boot
Password recovery
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
3850 Image Naming Convention
23
cat3k_caa-universalk9.SPA.03.02.01.SE.150-1.EX1.bin
cat3k = Platform Family
C = Converged
A = Access
A = Access Switch
Feature Set
Enabling/Disabling of
features controlled by
installed license
S = Digitally signed Image
P = Production Image
A = Key Version
IOS XE Version IOSd Version
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Booting IOS XE Software
Install Boot (default mode)
• Packages are installed on flash
• Supports AP image pre-download
• No additional memory requirement
• Image must be installed in flash:
• boot flash:packages.conf
Bundle Boot
• Packages are expanded in RAM
• No support for AP image pre-
download
• Additional memory equal to the size
of image bundle required
• Image can be booted from flash:,
usbflash: or tftp:
• boot flash:cat3k_caa-
universalk9.SPA.03.02.01.SE.1
50-1.EX1.bin
24
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
IOS XE Packages
Package Name Contents
cat3k_caa-base Kernel distribution and Open Source Software
cat3k_caa-infra IOS XE infrastructure software – Installer, HA
manager, etc.
cat3k_caa-iosd-universalk9 IOSd Software
cat3k_caa-platform Platform software – FED, Stack Manager, etc.
cat3k_caa-drivers Platform Drivers and UADP libraries
cat3k_caa-wcm Wireless Controller Module software, AP images
and Wireless Web GUI 25
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Common Bootup Issues
Switch does not auto-boot when rebooted after saving config
– Caused by CSCue76684
– Upgrade to software version 3.2.1SE or later
– Workaround 1: At the “switch:” prompt enter “boot flash:packages.conf”
– Workaround 2: Configure “boot system switch all flash:packages.conf” under the global config mode before saving config
Switch does not auto-boot when rebooted after configuring “boot system …”
– Root Cause: “boot config” is configured in startup-config
– Workaround: Remove “boot config …” from startup-config
– Fixed in software version 3.2.2SE
26
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
• Password recovery on 3850 does NOT follow the 3750 family procedure
• 3850 password recovery is as follows:
1. Power cycle switch and hold the Mode button (on the front top left) for a few seconds (officially 12) until the status LED gets amber, that will get you in Boot Loader prompt (Switch:)
1. Set the following variables
2. Boot the 3850
3850 Password recovery
Switch: SWITCH_IGNORE_STARTUP_CFG=1
Switch: SWITCH_DISABLE_PASSWORD_RECOVERY=0
Switch:
Switch: boot
Warning!
Console:9600 baud
8 data bits, No flow control
1 stop bit, No parity
27
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
4. Skip the initial configuration dialog and go to enable (no password required):
5. Copy startup-config back to running-config:
3850 Password recovery
--- System Configuration Dialog ---
Would you like to enter the initial configuration dialog? [yes/no]: no
Would you like to terminate autoinstall? [yes]: yes
Press RETURN to get started!
Switch> enable
Switch#
Switch# copy startup-config running-config
28
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
6. Go to global configuration, and remove or change the password:
7. Go to exec mode and enable reading of startup-config
8. Disable password recovery if required
3850 Password recovery
Switch# configure terminal
Switch(config)# no enable password
Switch(config)# no enable secret
Switch(config)# enable secret cisco
Switch(config)# end
Switch# write (copy running-config startup-config)
Switch# no system ignore startupconfig switch all
Switch# system disable password recovery switch all
29
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Software Upgrade on 3850
30
Software upgrade in Installed Mode is done via the “software install …” command
Prerequisites for software installation:
The switch’s free memory must be greater than the size of the bundle being installed
The free space in flash: must be greater than the size of the bundle being installed
All switches must be running in installed mode
When installing a bundle from a local storage device, the device must exist on all switches performing
the installation operation
The packages in the bundle to be installed must have valid digital signatures
A failed installation might require a rollback using “software rollback” command or a
manual clean using “software clean” command.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Upgrade/Install a Bundle on flash
Switch# software install file flash:cat3k_caa-universalk9.SPA.03.02.01.SE.150-1.EX1.bin Preparing install operation ... [2]: Copying software from active switch 2 to switch 1 [2]: Finished copying software to switch 1 [1 2]: Starting install operation [1 2]: Expanding bundle flash:cat3k_caa-… [1 2]: Copying package files [1 2]: Package files copied [1 2]: Finished expanding bundle flash:cat3k_caa-… [1 2]: Verifying and copying expanded package files to flash: [1 2]: Verified and copied expanded package files to flash: [1 2]: Starting compatibility checks [1 2]: Finished compatibility checks [1 2]: Starting application pre-installation processing [1 2]: Finished application pre-installation processing [1]: Old files list: Removed cat3k_caa-base.SPA.03.02.00.SE.pkg Removed cat3k_caa-drivers.SPA.03.00.00.SE.pkg Removed cat3k_caa-infra.SPA.03.02.00.SE.pkg Removed cat3k_caa-iosd-universalk9.SPA.150-1.EX.pkg Removed cat3k_caa-platform.SPA.03.02.00.SE.pkg Removed cat3k_caa-wcm.SPA.03.02.00.SE.pkg
Preparation stage
Installing to Flash
Post Install Checks
Removing old files
31
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Software Rollback Use the ‘software rollback’ command to revert to the previously installed package set (packages.conf.00-).
Switch# software rollback Preparing rollback operation ... [2]: Starting rollback operation [2]: Starting compatibility checks [2]: Finished compatibility checks [2]: Starting application pre-installation processing [2]: Finished application pre-installation processing [2]: Old files list: Removed cat3k_caa-base.SPA.03.02.01.SE.pkg Removed cat3k_caa-drivers.SPA.03.02.01.SE.pkg Removed cat3k_caa-infra.SPA.03.02.01.SE.pkg Removed cat3k_caa-iosd-universalk9.SPA.150-1.EX1.pkg Removed cat3k_caa-platform.SPA.03.02.01.SE.pkg Removed cat3k_caa-wcm.SPA.03.02.01.SE.pkg [2]: New files list: Added cat3k_caa-base.SPA.03.02.00.SE.pkg Added cat3k_caa-drivers.SPA.03.02.00.SE.pkg Added cat3k_caa-infra.SPA.03.02.00.SE.pkg Added cat3k_caa-iosd-universalk9.SSA.150-1.EX.pkg Added cat3k_caa-platform.SPA.03.02.00.SE.pkg Added cat3k_caa-wcm.SPA.03.02.00.SE.pkg [2]: Creating pending provisioning file [2]: Finished rolling back software changes. New software will load on reboot. [2]: Do you want to proceed with reload? [yes/no]: n Switch#
Removed newly installed image
Reverted to older image
32
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Recover a Corrupted Install
33
Copy the image bundle to USB flash and bootup using the following command from
the Bootloader prompt:
boot usbflash0:cat3k_caa-universalk9.SPA.03.02.01.SE.150-1.EX1.bin
Copy the image bundle to USB flash and recover the switch by using the recovery
mechanism built into the switch from the Bootloader prompt:
emergency-install usbflash0:cat3k_caa-
universalk9.SPA.03.02.01.SE.150-1.EX1.bin
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Memory and CPU
Installer and Bootup
Licensing
Hardware Troubleshooting
Agenda Architecture & Product Overview
Multicast Forwarding
Unicast Forwarding
CPU Punt Path
Stacking
High Availability
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
35
Right To Use (RTU) Licensing on the 3850
How to resolve license mismatch in stack
RTU Licensing show commands
Wireless AP Licensing
Wireless License Behavior in stack
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Right To Use (RTU) / Honor Based Licensing
Trust Based Licensing Model
Built in licenses, not tied to Unique Device Identifier
Three license levels – lanbase, ipbase and ipservices
Activated using CLI by accepting the End User License Agreement
Portable across devices
No Need to access cisco.com License Portal
36
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
License Mismatch
37
license right-to-use deactivate ipservices
license right-to-use activate ipbase acceptEULA
Reload switch
IP Base
IP Base
IP Base IP
Services
A
S
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Licensing Show commands
38
Switch# show license right-to-use slot 1
Slot# License name Type Count Period left
----------------------------------------------------------
1 ipbase permanent N/A Lifetime
1 lanbase permanent N/A Lifetime
1 apcount adder 4 Lifetime
License Level on Reboot: ipservices
Switch# show license right-to-use mismatch
Slot# License Name Adder AP Count Base AP Count
---------------------------------------------------------------
3 ipservices 0 0
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Wireless Licenses
50 AP Licenses can be applied on a 3850 stack
Licenses can be applied on individual switches or on the Active
Licenses can be applied in increments of 1
AP adder licenses are applied using the CLI: license right-to-use activate apcount 10 slot 1 acceptEULA
AP licenses are applied on the Mobility Controller
39
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Wireless License Behavior
40
4 AP
License
S
A
A
No License. AP
unable to join.
Active loses power
S
Recommendation is to apply licenses on each switch based on the number of APs connected to the switch.
license right-to-use activate apcount 2 slot 1
license right-to-use activate apcount 2 slot 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Licensing
CPU Punt Path
Memory and CPU
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
Multicast Forwarding
Unicast Forwarding
Stacking
High Availability
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
42
3850 CPU Complex
Troubleshooting memory utilization
Troubleshooting high CPU utilization
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
CPU Complex
43
Cavium 6230
800 MHz, 4 core CPU
2MB L2 Cache
UADP 1
UADP 2
USB/RJ-45 Console 10/100/1000 RJ-45
Ethernet Mgmt
SGMII UART
PCIe
PCIe
4GB DDR3
w/ ECC
DDR3 - 1333
FPGA for
Stack Power
I2C
RTC
ACT II
FPGA for PHY,
LED, etc.
I2C
2GB Flash 64MB
Bootloader
Boot Bus
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
High Memory Utilization Frequently Asked Questions
44
Why should I be concerned about high memory utilization ? It is very important have enough free memory to support features and network convergence events that require
transient memory.
What are the usual symptoms of high memory usage ?
Memory utilization of process(es) keeps increasing
System runs out of buffers and software packet forwarding stops
Memory allocation failures are reported
System crashes after reporting out of memory
At what percentage level should I start troubleshooting ? It depends on the nature and level of feature config on the switch. It is very essential to find a baseline memory
usage during normal working conditions, and start troubleshooting when it goes above specific threshold.
E.g., Baseline memory usage 40%. Start troubleshooting when the memory goes above 60% and constantly keeps
increasing without adding any new configuration.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Switch1# show processes memory sorted
System memory : 3930916K total, 1118032K used, 2812884K free, 221968K kernel reserved
Lowest(b) : 2252987972
PID Text Data Stack Heap RSS Total Process
10623 56892 36452 92 5400 196116 336728 iosd
5534 8716 311168 92 4620 136908 562460 fed
10619 21976 555372 88 13980 102320 723240 wcm
6032 4 97708 116 91996 99044 116676 idope.py
12259 4 193244 236 38244 73672 299464 wnweb_paster.py
5536 660 163524 88 4332 55968 336496 stack-mgr
6057 3532 137308 88 2200 54200 311676 ffm
6076 112 160908 88 6764 44728 233548 cli_agent
6058 1232 287972 88 8112 38352 438040 eicored
Memory show commands
45
Total Memory IOS-XE Process
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Memory show commands
46
Switch1# show processes memory detailed process iosd sorted
Processor Pool Total: 536870912 Used: 135242980 Free: 401627932
IOS Proce Pool Total: 16777216 Used: 9483360 Free: 7293856
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 168268072 31876024 126376204 0 0 *Init*
164 0 1534944 0 1558112 907264 0 NGWC DOT1X Proce
0 0 0 0 984492 0 0 *MallocLite*
1 0 657344 1544 678968 0 0 Chunk Manager
276 0 925564 297800 563696 0 0 os_info_p provid
39 0 415892 1856 376480 0 0 IPC Seat RX Cont
250 0 298204 464 320908 0 0 IPC LC Message H
IOS Tasks
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Common Causes for high memory utilization
Common Cause Recommended Solution
Extensive Config Reduce configuration to supported scale
Excessive memory allocated to trace buffers Reset trace buffers to default sizes
DoS Attack/Punted traffic causing buffer
depletion
Identify packets and block them using an
ACL
Protocol flaps/re-convergence causing high
transient memory utilization
Identify reason for network instability
Memory Leak caused by software bug Open a Service Request
47
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Troubleshooting Memory
Troubleshooting Steps Commands
Check memory usage on system show processes memory sorted
Check memory usage of a particular process show processes memory detailed process fed
Check memory usage of IOSd show processes memory detailed process iosd
Check allocators of memory within IOSd show memory detailed process iosd allocating-
process totals
Command Summary
48
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
High CPU Utilization Frequently Asked Questions
49
Why should I be concerned about high CPU utilization ? It is very important to protect the control plane for network stability, as resources (CPU, Memory and buffer) are
shared by control plane and data plane traffic (sent to CPU for further processing)
What are the usual symptoms of high CPU usage ?
Control plane instability e.g., OSPF flap
Reduced switching / forwarding performance
Slow response to Telnet / SSH
SNMP poll miss
At what percentage level should I start troubleshooting ? It depends on the nature and level of the traffic. It is very essential to find a baseline CPU usage during normal
working conditions, and start troubleshooting when it goes above a specific threshold.
E.g., Baseline CPU usage 25%. Start troubleshooting when the CPU usage is consistently at 50% or above.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Switch# show proc cpu sorted
Core 0: CPU utilization for five seconds: 96%; one minute: 7%; five minutes: 6%
Core 1: CPU utilization for five seconds: 5%; one minute: 1%; five minutes: 1%
Core 2: CPU utilization for five seconds: 0%; one minute: 0%; five minutes: 0%
Core 3: CPU utilization for five seconds: 41%; one minute: 1%; five minutes: 1%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5533 120300 1608989 74 0.29 0.40 0.42 1088 fed
5535 44890 1401868 32 0.24 0.11 0.10 0 stack-mgr
10582 416280 5787047 71 34.25 0.57 0.62 34816 iosd
6201 111520 119850 930 0.15 0.15 0.15 0 cpumemd
5534 38430 3608873 10 0.10 0.10 0.10 0 platform_mgr
10578 115030 4737397 24 0.10 0.12 0.11 0 wcm
5455 1500 40856 36 0.05 0.05 0.05 1088 slproc
6183 5270 211347 24 0.05 0.02 0.04 0 obfld
6185 4320 110250 39 0.05 0.01 0.03 0 console_relay
6198 20900 186795 111 0.05 0.02 0.00 0 ffm
1 1700 1112 1528 0.00 0.09 1.43 0 init
2 0 138 0 0.00 0.00 0.00 0 kthreadd
3 10 1634 6 0.00 0.00 0.00 0 migration/0
4 0 3 0 0.00 0.00 0.00 0 sirq-high/0
Troubleshooting High CPU
50
Quad Core CPU
Platform Processes
IOS-XE Processes
137% across 4
cores
Identify the Culprit
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Troubleshooting High CPU
51
Switch# show processes cpu detailed process iosd sorted
Core 0: CPU utilization for five seconds: 96%; one minute: 7%; five minutes: 6%
Core 1: CPU utilization for five seconds: 5%; one minute: 1%; five minutes: 1%
Core 2: CPU utilization for five seconds: 0%; one minute: 0%; five minutes: 0%
Core 3: CPU utilization for five seconds: 41%; one minute: 1%; five minutes: 1%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
10582 L 451160 6379641 70 34.25 0.71 0.60 34816 iosd
10582 L 0 10582 414060 6194757 0 24.00 0.60 0.50 34816 iosd
10582 L 3 11543 36980 180107 0 10.25 0.11 0.10 0 iosd.fastpath
10582 L 2 11544 120 4777 0 0.00 0.00 0.00 34816 iosd.aux
6 I 57680 5216 0 3.00 0.33 0.22 0 Check heaps
304 I 2200 1790 0 12.17 0.00 0.00 0 HTTP CORE
218 I 2370 14495 0 8.33 0.00 0.00 0 IP Input
211 I 190 214 0 0.33 0.00 0.00 0 RSMP Server
306 I 10 23 0 0.11 0.00 0.00 0 SEP NODE PROC
5 I 0 2 0 0.00 0.00 0.00 0 IPC ISSU Dispatch P
7 I 220 336 0 0.00 0.00 0.00 0 Pool Manager
3 I 0 1 0 0.00 0.00 0.00 0 HA-IDB-SYNC
High CPU caused
by HTTP traffic
Interrupt switched
traffic (eg. Wireless
Control)
Drill Down Deeper
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Troubleshooting High CPU
Troubleshooting Steps Commands
Check CPU usage on IOS threads show process cpu detailed process iosd
[sorted]
Check CPU usage on platform dependent and
Nova threads
show process cpu detailed process {fed |
platform_mgr | stack-mgr | ha_mgr | eicored…}
Check traffic on the RX and TX CPU queues show platform punt client, show platform punt tx
Check details of CPU queues show platform punt statistics port-asic 0 cpuq 0
direction {rx | tx}
Command Summary
52
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Stacking
Memory and CPU
Licensing
CPU Punt Path
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
Multicast Forwarding
Unicast Forwarding
High Availability
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
54
3850 CPU Punt Path
Reasons for punting packets to CPU
Identifying and capturing packets punted to CPU
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
CPU Punt Path Architecture
55
IOSd WCM Punt Shim
Forwarding Engine Driver
Packet Handler
UADP ASIC
32 RX PDS Queues 8 TX PDS Queues
32 RX Queues 8 TX Queues
Processes
Control Packets
Processes
Wireless Control
Packets
Interfaces with
UADP ASIC and
Packet Delivery
Service (PDS)
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Common Cause for Punting Traffic to CPU
Common Cause Recommended Solution
Same interface forwarding change design, use “no ip redirect”
ACL logging disable ACL logging
ACL deny causing switch to send ICMP
unreachable
no ip unreachables1
Forwarding/Feature exception (out of TCAM/adj
space)
reduce TCAM usage
resize TCAM region (TCAM2/3)
SW-supported feature disable the feature or reduce the amount of
traffic
IP packets with TTL<2 or options disable the offending traffic
Broadcast Storm Fix STP loop, disable traffic
Unexpected control/data traffic Control Plane Policing (CoPP), Deny ACL
Software Bug Open a Service Request
1. Should be configured on all the L3 interfaces of the switch.
56
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Decoding CPU Queues
57
Switch# show platform punt client
tag buffer jumbo fallback packets received failures
alloc free bytes conv buf
65536 0/1024/1600 0/0 0/512 64845 64845 3371071 0 0
65544 0/ 96/1600 0/4 0/0 0 0 0 0 0
65545 0/ 96/1600 0/8 0/32 1947 1947 612588 0 0
65546 0/ 512/1600 0/32 0/512 13563 137795 24587306 0 0
65548 0/ 512/1600 0/32 0/256 10903 10903 650232 0 0
65551 0/ 512/1600 0/0 0/256 56 56 12088 0 0
65561 411/ 512/1600 0/0 0/128 557245 556834 39010862 0 0
65562 0/ 512/1600 0/16 0/256 0 0 0 0 0
CPU Queue Number
25 (65561-65536) Number of packets in
queue awaiting
processing
Size of Queue Size of each buffer
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Displaying packets in the queue
show buffers detailed process iosd assigned packet | beg ng3k_rx25
Buffer information for ng3k_rx25 buffer at 0x35E98E8C
data_area 0x35E9932C, refcount 1, next 0x0, flags 0x80
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input Vlan10, if_output 0x0 (None)
source: 10.32.111.83, destination: 10.33.21.219, id: 0x4BE0, ttl: 63,
TOS: 0 prot: 6, source port 51378, destination port 22
35E99382: 6400F124 F1C11410 9FE43A49 08004500 d.q$qA...d:I..E.
35E99392: 00984BE0 40003F06 56110A20 6F530A21 ..K`@.?.V.. oS.!
35E993A2: 15DBC8B2 0016588A DB9F6C34 421A5018 .[H2..X.[.l4B.P.
35E993B2: FFFF8666 000072A2 E1AB5431 78970F84 ...f..r"a+T1x...
58
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
CPU Punt Path
High Availability
Stacking
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
Multicast Forwarding
Unicast Forwarding
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
60
3850 Stacking Architecture
Stacking Show commands
Troubleshooting failure to form a stack
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
3850 StackWise-480 Overview
• 3850 StackWise-480 is a new generation of Catalyst 3850 stacking
240Gbps of bandwidth (120Gbps TX & 120Gbps RX per connector)
Similar to previous stacking implementations, ring redundancy is achieved via ring-wrap capabilities provided in hardware
NOT backward compatible with currently fielded stacking technologies, most notably StackWise Plus
61
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
3850 StackWise-480 Cables
• StackWise-480 currently supports 3 cables
STACK-T1-50CM = 0.5m cable
STACK-T1-1M = 1m cable
STACK-T1-3M = 3m cable
• All StackWise-480 cables include ACT II chips for counterfeit protection
62
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
• 6 rings in total
• 3 rings go East
• 3 rings go West
• Each ring is 40G
• Total Stack BW = 240G
• With Spatial Reuse = 480G
Stack Interface of UADP
Stack Interface of UADP ASIC
Assuming 4 x 24-port 3850 Switches
Packets are segmented/reassembled in HW (256 byte
segments)
Understanding the Stack Ring
Is math really an
opinion?
63
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Destination Stripping Packet travels ½ the rings. Taken out of stack by destination
1 3
1 3
Assuming 4 x 24-port 3850 Switches
4 2
4 2
Spatial Reuse
64
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Show commands
65
Switch# show switch detail
Switch/Stack Mac Address : 6400.f124.df80 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 6400.f124.df80 10 0 Ready
2 Standby 6400.f124.de80 1 0 Ready
Stack Port Status Neighbors
Switch# Port 1 Port 2 Port 1 Port 2
--------------------------------------------------------
1 OK OK 2 2
2 OK OK 1 1
Priority, followed by MAC
Address determines which
switch gets elected as Active
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Show commands
66
Switch# show switch stack-ports summary
Sw#/Port# Port Status Neighbor Cable Length Link OK Link Active Sync OK #Changes to LinkOK In Loopback
---------------------------------------------------------------------------------------------------------------
1/1 OK 2 50cm Yes Yes Yes 0 No
1/2 OK 2 Unknown Yes Yes Yes 0 No
2/1 OK 1 100cm Yes Yes Yes 1 No
2/2 OK 1 50cm Yes Yes Yes 1 No
Cable with corrupted
EEPROM
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Image Version Mismatch
If the switches are in version mismatch state, they will not stack
Debugging:
– Check if the switch version matches the active using show version command
– If they do not match, upgrade the switch to the Active’s version
Switch# show switch
Switch# Role Mac Address Priority Version State
---------------------------------------------------------------------------
*1 Active 6400.f125.1480 1 V01 Ready
2 Member 6400.f125.2680 1 0 V-Mismatch
3 Member 6400.f125.2500 1 0 V-Mismatch
4 Member 6400.f125.2480 1 0 V-Mismatch
67
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Switch Stuck in Syncing State
If a switch is stuck in syncing state
Debugging:
– Run the command “sh switch” to see the states
– Open a Service Request with Cisco TAC
Switch# show switch
Switch# Role Mac Address Priority Version State
---------------------------------------------------------------------------
*1 Active 6400.f125.1480 1 V01 Ready
2 Standby 6400.f125.2680 1 V01 Ready
3 Member 6400.f125.2500 1 V01 Ready
4 Member 6400.f125.2480 1 0 Syncing
68
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Stacking
Unicast Forwarding
CPU Punt Path
High Availability
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
Multicast Forwarding
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
70
3850 HA Architecture
Election of Active and Standby
Show commands for checking HA states
Troubleshooting
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Catalyst 3750-X – StackWise-Plus - Hybrid control-plane processing
- N:1 stateless control-plane redundancy
- Distributed L2/L3 Forwarding Redundancy
- Stateless L3 protocol Redundancy
Catalyst 3850 – StackWise-480 - Centralized control-plane processing
- 1+1 Stateful redundancy (SSO)
- Distributed L2/L3 Forwarding Redundancy
- IOS HA Framework alignment for L3 protocol
HA Redundancy – Shift from 3750-X
71
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
HA SSO Architecture
72
Interfaces
L2
L3
QoS
Interfaces
L2
L3
QoS
Wireless
Wireless
Feature State is synced between Active and Standby Member in stack
Feature States are inactive on Standby Member
S
A
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Route Processor Domain – a set of SW processes (e.g. IOSd, WCM) that
implement the centralized Active and Standby portions of the stack control plane
Line Card Domain – a set of SW processes (e.g. FED, Platform Manager) that
implement the distributed Line Card portions of the stack control plane
Infra Domain – Support SW for the RP and LC Domains
Active Switch – supports the Active RP Domain, a LC Domain and Infra Domain
Standby Switch – supports the Standby RP Domain, a LC Domain and Infra
Domain
Member Switch – supports a LC Domain and Infra Domain.
Election – assigning roles or functions within the stack
Software HA Processes– Roles and Definitions
73
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
RP Infra LC
RP Infra
Infra LC
Infra LC
S LC
• Active starts RP Domain
(IOSd, WCM, etc) locally
• Programs hardware on all LC Domains
• Traffic resumes once hardware is
programmed
• Starts 2min Timer to elect Standby in parallel
• Active elects Standby
• Standby starts RP Domain locally
• Starts Bulk Sync with Active RP
• Standby reaches “Standby Hot”
2min timer
A
Catalyst 3850 – HA State Machine
74
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Switch# show switch
Switch/Stack Mac Address : 2037.06cf.0e80
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 2037.06cf.0e80 10 PP Ready
2 Standby 2037.06cf.3380 8 PP Ready
3 Member 2037.06cf.1400 6 PP Ready
4 Member 2037.06cf.3000 4 PP Ready
* Indicates which member is providing the “stack Identity” (aka “stack MAC)
Active
Standby
Show switch with SSO
75
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Show redundancy states
76
Switch# show redundancy states my state = 13 –ACTIVE
peer state = 8 -STANDBY HOT
Mode = Duplex
Unit ID = 2
Redundancy Mode (Operational) = SSO
Redundancy Mode (Configured) = SSO
Redundancy State = SSO
Manual Swact = enabled
Communications = Up
client count = 76
client_notification_TMR = 360000 milliseconds
keep_alive TMR = 9000 milliseconds
keep_alive count = 0
keep_alive threshold = 9
RF debug mask = 0
Terminal state for SSO. If “peer
state” is stuck in any other state
for more than 10 minutes, open
a service request with TAC
If Communication channel is not
Up, there might be a problem
with stack connectivity. Check
stack cable.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
High Availability
Multicast Forwarding
Stacking
Unicast Forwarding
CPU Punt Path
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
Wireless
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
78
Examine Hardware resources in use by each feature
Commands to debug Unicast L3 Forwarding on the 3850
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Configuration and Show commands
No new configuration added for Unicast Features on 3850
Configuration, show commands compatible with 3750X
Additional Platform CLIs have been added
Refer to configuration guide and command line reference for full details
79
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
TCAM Utilization
80
Switch1# show platform tcam utilization asic all
CAM Utilization for ASIC# 0
Table Max Values Used Values
--------------------------------------------------------------------------
Unicast MAC addresses 32768/512 82/22
Directly or indirectly connected routes 32768/8192 7/89
IGMP and Multicast groups 8192/512 0/16
Security Access Control Entries 3072 173
QoS Access Control Entries 2816 52
Netflow ACEs 1024 15
Input Microflow policer ACEs 256 7
Output Microflow policer ACEs 256 7
Control Plane Entries 512 187
Policy Based Routing ACEs 1024 9
Tunnels 256 12
Input Security Associations 256 4
Output Security Associations and Policies 256 9
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Unicast L3 Forwarding Show Commands
81
Switch1# show ip cef 72.163.4.161 internal
0.0.0.0/0, epoch 3, flags default route, RIB[S], refcount 6, per-destination sharing
sources: RIB, DRH
feature space:
Broker: linked, distributed at 1st priority
ifnums:
GigabitEthernet1/0/1(72): 172.25.33.1
path 32733FA4, path list 38B46BB8, share 1/1, type recursive, for IPv4
recursive via 172.25.33.1[IPv4:Default], fib 38B47A30, 1 terminal fib, v4:Default:172.25.33.1/32
path 32733F34, path list 36F8C660, share 1/1, type adjacency prefix, for IPv4
attached to GigabitEthernet1/0/1, adjacency IP adj out of GigabitEthernet1/0/1, addr 172.25.33.1
37BCEF40
output chain: IP adj out of GigabitEthernet1/0/1, addr 172.25.33.1 37BCEF40
Switch1#
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Unicast L3 Forwarding Show Commands
82
Switch1# show adjacency gi1/0/1 172.25.33.1 detail
Protocol Interface Address
IP GigabitEthernet1/0/1 172.25.33.1(11)
0 packets, 0 bytes
epoch 0
sourced in sev-epoch 1
Encap length 14
0011922DA3416400F124DFE40800
L2 destination address byte offset 0
L2 destination address byte length 6
Link-type after encap: ip
ARP
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Platform Show Commands
83
Switch1#show platform ip adj
IP Adj entries
dest if_name dst_mac si_hdl ri_hdl pd_flags token
---- ------- ------- ------ ------ -------- -----
172.25.33.55 Gi1/0/1 0050.56ba.0053 0x4f0580f0 0x4f051f20 0x51 0xdf
172.25.33.1 Gi1/0/1 0011.922d.a341 0x4f051210 0x4f058ff8 0x51 0xe6
Switch1#show platform abstraction print-resource-handle 0x4f058ff8 0
Handle:0x4f058ff8 Res-Type:ASIC_RSC_RI Res-Switch-Num:255 Asic-Num:255 Feature-
ID:AL_FID_L3_UNICAST_IPV4 Lkp-ftr-id:LKP_FEAT_INVALID ref_count:1
Hardware Indices/Handles:priv_ri/priv_si Handle:0x4f051210 index0:0x13 mtu_index/l3u_ri_index0:0x0
Features sharing this resource:55 (1)
Cookie length 56
0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x3 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x11 0x92
0x2d 0xa3 0x41 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0
0x0 0x0 0x0 0x0 0x0 0x0 0x0
==============================================================
Rewrite Index Handle
Destination MAC Address
Multiple Features can share RI
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Wireless
Unicast Forwarding
High Availability
Multicast Forwarding
Stacking
CPU Punt Path
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
QoS
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
85
Multicast display and configuration commands on the 3850
Platform show commands for IGMP Snooping
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Configuration & Display
IP igmp snooping is enabled by default.
IPv6 mld snooping is enabled using‘ipv6 mld snooping’ command
No platform configuration commands.
Some of the IOS configuration and display commands are listed below.
Refer IOS Command Manual for a complete list of commands.
Configuration Commands:
ip igmp snooping ip igmp snooping querier ip igmp snooping vlan <num> mrouter interface <interface-name> ip igmp snooping vlan <num> static <x.x.x.x> interface <interface-name> ipv6 mld snooping ipv6 mld snooping vlan <num> mrouter interface <interface-name> ipv6 mld snooping vlan <num> static <x:x:x::x> interface <interface-name>
Display Commands:
show ip igmp snooping show ip igmp snooping mrouter show ip igmp snoop querier show ipv6 mld snooping address show ipv6 mld snooping mrouter show ipv6 mld snooping querier
86
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Platform Display
Switch#show platform ip igmp snooping ? <1-4094> VLAN ID groups Display all IGMP multicast group information retry retry information switch Switch number vlan vlan information | Output modifiers <cr>
Display platform information for igmp snooping group entry Switch#show platform ip igmp snoop 1 group 226.1.1.1 L2 Multicast Group Information ============================== Vlan Id: 1 Group: 226.1.1.1 Member ports: Gi1/0/6 (36906207297929242) CAPWAP ports: Flags: Host Type Flags: 0 Failure Flags : 0 DI handler: 0x61c1b248 REP RI handler: 0x61c1b400 SI handler: 0x61c1b730 TCAM handler: 0x61c1b7d8 Group Key: 00000001E2010101000000000000000000000000
Receiver Ports for Group,
interface-name(interface-id)
If Entry is installed successfully in
asic, Failure Flags will be zero
Resource Handles for entry in ASIC, If it is
non-zero, Resource is allocated successfully
87
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
QoS
Multicast Forwarding
Wireless
Unicast Forwarding
High Availability
Stacking
CPU Punt Path
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
Misc. Tools and Tricks
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
89
Wireless Converged Access Components and Architecture
State Transitions a client goes through
Troubleshooting a wireless roam
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
• Can act as a Mobility Agent (MA) for terminating CAPWAP tunnels for locally connected APs …
• as well as a Mobility Controller (MC) for other Mobility Agent (MA) switches, in small deployments - MA/MC functionality works on a Stack of Catalyst 3850 Switches - MA/MC functionality runs on Stack Master - Stack Standby synchronizes some information (useful for intra-stack HA)
Best-in-Class Wired Switch – with Integrated
Wireless Mobility functionality
Converged Access –
Physical Entities – Catalyst 3850 Switch Stack
MC
MA
90
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Converged Access –
Traffic Flow and Roaming – Branch, L2 / L3 Roam (within SPG)
SPG
MC MA MA
1142-1
10.101.1.109
10.101.3.109
MA
PoA
PoP
L09-3850s-3# show wireless client summary
Number of Local Clients : 1
MAC Address AP Name WLAN State Protocol
--------------------------------------------------------------------------------
001e.65b7.7d1a L09-AP1142-1 2 UP 11n(5)
L09-3850s-3# show wcdb database all
Total Number of Wireless Clients = 1
Local Clients = 1
Mac Address VlanId IP Address Src If Auth Mob
-------------- ------ --------------- ------------------ -------- -------
001e.65b7.7d1a 2003 10.101.203.1 0x009350C0000000E4 RUN LOCAL
L09-3850s-3# show wireless client mac 001e.65b7.7d1a detail
Client MAC Address : 001e.65b7.7d1a
Mobility State : Local
Client States:
L2_AUTH
AUTHENTICATING
LEARN_IP
RUN
91
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Converged Access –
Traffic Flow and Roaming – Branch, L2 / L3 Roam (within SPG)
SPG
1142-1
10.101.3.109
MC MA MA
10.101.1.109
MA
PoA
PoP
Ca5
L09-3850s-3# show mac address dynamic | inc Ca
2003 001e.65b7.7d1a DYNAMIC Ca5
L09-3850s-3# show capwap summary
Name APName Type PhyPortIf Mode McastIf
------ -------------------------------- ---- --------- --------- -------
Ca1 - mob - unicast -
Ca2 - mob - unicast -
Ca5 L09-AP1142-1 data Gi1/0/7 multicast Ca4
Name SrcIP SrcPort DestIP DstPort DtlsEn MTU
------ --------------- ------- --------------- ------- ------ -----
Ca1 10.101.3.109 16667 10.101.1.109 16667 No 1464
Ca2 10.101.3.109 16667 10.101.2.109 16667 No 1464
Ca5 10.101.3.109 5247 10.101.3.98 31901 No 1449
L09-3850s-3# show ip dhcp snooping binding
MacAddress IpAddress Lease(sec) Type VLAN Interface
------------------ --------------- ---------- ------------- ---- --------------------
00:1E:65:b7:7d:1a 10.101.203.1 10617 dhcp-snooping 2003 Capwap5
92
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Converged Access –
Traffic Flow and Roaming – Branch, L2 / L3 Roam (within SPG)
SPG
1142-1
Ca5
3600-1
L09-3850-1# show wireless client summary
MAC Address AP Name WLAN State Protocol
--------------------------------------------------------------------------------
001e.65b7.7d1a 3600-1 2 UP 11n(5)
L09-3850-1# show wcdb database all
Total Number of Wireless Clients = 1
Foreign Clients = 1
Mac Address VlanId IP Address Src If Auth Mob
-------------- ------ --------------- ------------------ -------- -------
001e.65b7.7d1a 2001 10.101.203.1 0x00C55A40000000A6 RUN FOREIGN
10.101.3.109
SPG
MC MA MA
10.101.1.109
MA
PoP
PoA
93
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Wireless
Misc. Tools and Tricks
Multicast Forwarding
Unicast Forwarding
High Availability
Stacking
CPU Punt Path
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
QoS
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
In this section, you will learn about …
95
QoS Features available on the 3850
MQC Configuration commands
Platform QoS commands
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Modular QoS based CLI (MQC)
– Alignment with 4500E series (Sup6, Sup7)
– Class-based Queuing, Policing, Shaping, Marking
More Queues
– Up to 2P6Q3T queuing capabilities
– Standard 3750 provides 1P3Q3T
– Not limited to 2 queue-sets
– Flexible MQC Provisioning abstracts queuing hardware
Granular QoS control at the wireless edge
Tunnel termination allows customers to provide QoS treatment per SSIDs, per-Clients and common treatment of wired and wireless traffic throughout the network
Enhanced Bandwidth Management
Approximate Fair Drop (AFD) Bandwidth Management ensures fairness at Client, SSID and Radio levels for NRT traffic
Wireless Specific Interface Control
Policing capabilities Per-SSID, Per-Client upstream and downstream
AAA support for dynamic Client based QoS and Security policies
Per SSID Bandwidth Management
Wired Wireless
QoS – What’s New with Converged Access Policy-map PER-PORT-POLICING
Class VOIP
set dscp ef
police 128000 conform-action transmit
exceed-action drop
Class VIDEO
set dscp CS4
police 384000 conform-action transmit
exceed-action drop
Class SIGNALING
set dscp cs3
police 32000 conform-action transmit
exceed-action drop
Class TRANSACTIONAL-DATA
set dscp af21
Class class-default
set dscp default
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Platform QoS CLI
97
Switch# show platform qos policy target GigabitEthernet 1/0/48
Input policy :
--------------
Not attached
Output policy :
--------------
POLICY: defportangn Num Classes:1 PLC Map Targets:0 Queue LBL Targets uplink:0 downlink:0
PMAP:0x6345d778 NextPMAP:0x585d5518 PrevPMAP:0x57b02b98
UP Mask: 0, Lookup Type:0
COS Mask: 0, dscp mask:0
Filter flags: 0, Action Flags:0x14, num_classmaps 1 policy_type: MARKING/POLICING
nfl_req_pending_cnt:0 pmap_qsize:0
CLASS: class-default
CMAP:0x124b42a0 Next:(nil) Prev:(nil)
Masks:- UP:0, CoS: 0, Dscp:0
Filter flags 0
Not Supported
Negate: NO Next:(nil) . . . .
Switch# sh platform qos policy hw_state target GigabitEthernet 1/0/48
Input policy : Not attached
Output policy : defportangn
H/W programming State: INSTALLED IN HW
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Platform Wireless QoS CLI
98
Switch# sh wireless client username naoshad
MAC Address AP Name WLAN State Protocol
--------------------------------------------------------------
2477.038f.24c4 AP-3600-1 4 UP 11n(5)
Switch# show platform qos wireless afd client mac 2477.038f.24c4
IF Type:CLIENT
ASIC: 0
Index: 2320
Afd Max Rate: 4194300
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
QoS
Summary
Wireless
Misc. Tools and Tricks
Multicast Forwarding
Unicast Forwarding
High Availability
Stacking
CPU Punt Path
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Additional Troubleshooting Commands
100
Topic Command
Platform specific Feature information show tech-support platform <feature> (eg.
wireless, acl, fnf, etc.)
Trace Buffers for non-IOSd processes show mgmt-infra trace messages <component>
(eg. fed-punject-detail, stack-mgr-events, etc.)
Generate Live Core of a Process (internal
command)
resource process dump <process id obtained from
show process> [ switch <switch number> ]
Generate System Report (internal command) resource create_system_report
Identify memory leaks show memory debug leaks detailed process
<process name> summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Core Dumps and System Reports
101
• System generates a fullcore, crashinfo and System Report when a process terminates abnormally
• A System Report is generated each time the switch is rebooted
• System Report contains a dump of all the trace buffers in the system
• When filing a TAC case, please attach the fullcore, crashinfo and System Report files (whatever is applicable) from the crashinfo: filesystem
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Wireless
Misc. Tools and Tricks
Multicast Forwarding
Unicast Forwarding
High Availability
Stacking
CPU Punt Path
Memory and CPU
Licensing
Installer and Bootup
Hardware Troubleshooting
Agenda Architecture & Product Overview
QoS
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Summary
• Provided a High Level Architectural overview of features on the 3850
• Basic Troubleshooting functionality available on the 3850
• Do you have a better understanding of:
• 3850 as a platform
• Key differences between 3850 and 3750X
• Basic troubleshooting on the 3850
• Would you like to see:
• More/Less of any particular topic
• More topics
• Longer session
103
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCRS-3146 Cisco Public
Maximize your Cisco Live experience with your
free Cisco Live 365 account. Download session
PDFs, view sessions on-demand and participate in
live activities throughout the year. Click the Enter
Cisco Live 365 button in your Cisco Live portal to
log in.
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Cisco Daily Challenge points for each session evaluation you complete.
Complete your session evaluation online now through either the mobile app or internet kiosk stations.
104