+ All Categories
Home > Documents > BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

Date post: 08-Jul-2018
Category:
Upload: hola-amigo
View: 217 times
Download: 0 times
Share this document with a friend

of 54

Transcript
  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    1/132

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    2/132

    Troubleshooting Wireless LANsBRKEWN-3011

    Patrick Croak

    Technical Leader

    CCIE Wireless #34712

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    3/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Wireless LANs

    Software and Support

    Troubleshooting Basics

     AP Discovery/Join

    WLC Config/Monitoring

    Client Connectivity

    Mobility

    Packet Analysis

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    4/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Software and Support

    Opening a TAC Service Request

    Cisco Support Model

    What to expect from TAC

    How does escalation work?

    WLC Software Trains

    CCO (ED/MD/AW)

    Engineering Specials

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    5/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    What should I have ready?

     – Clear problem description –  Always: Show run-config

     – If client involved, always: debug client  

     – Your analysis of any data provided

     – Set clear expectation of timeline and severity

    Software and SupportOpening a TAC Service Request

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    6/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Software and Support

    What to expect from TAC

     – Configuration assistance – Problem analysis / bug isolation

     – Workarounds or fixes

     –  Action plan to resolve SR

     – Hardware replacement

     – Engage BU when appropriate

    Cisco Support Model - Expectations

    What not to expect from

     ‒ Design and deployment

     ‒ Complete configuration

     ‒ Sales related informatio

     ‒ RF Tuning

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    7/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Software and Support

    TAC Escalation Process

     – Multi-Tier support resources within a technology – TAC to engage resources (TAC/BU) when appropriate

     – SR ownership might not change hands

    Customer Escalation Process

     – Raise SR priority (S1/S2)

     – Engage account team

     – Your satisfaction is important to the Cisco TAC. If you have concerns aboprogress of your case, please contact your regional TAC.

    Cisco Support Model - Escalation

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    8/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Software and Support

    CCO - Cisco.com release

     – 7.0.240.0, 7.3.112.0, 7.4.100.0, etc…  – Full test cycle

     – Classified as ED when posted

     AssureWave

     –  AW is no longer tagged on CCO, but AW validation results are available http://www.cisco.com/go/assurewave 

     – Results available 4 weeks after CCO

    MD

     – MD tag represents stable releases for mass adoption

     – MD tag will be considered on CCO after AW release validation, 10 weeksTAC/Escalation signoff

    WLC Software Trains - CCO

    http://www.cisco.com/go/assurewavehttp://www.cisco.com/go/assurewave

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    9/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Software and Support

    Not all images are created equally

    Diagnostic/Validation

     – Debug Image

     – Test Image

    Special Fix

    Production Ready

     – Escalation Code

     – Beta / Pre-Release

     – CCO

    WLC Software Trains - Engineering Special (ES)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    10/132© 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Basics

    Troubleshooting 101

     – Clearly define the problem – Understand any possible triggers

     – Know the expected behavior

     – Reproducibility

    Recommended Tools

     – Spectrum Analyzer

     – Wireless Sniffer and Wired Captures

    Prob

    Defin

    Ques

    Te

    Solut

    Ana

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    11/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Basics

    Troubleshooting is an art with no right or wrong procedure, but bes

    logical methodology. Step 1: Define the problem

     – It is crucial to understand all possible details of a problem

     – Knowing what is and is not working will go a long way

     – With a proper understanding of the problem description you can skip ma

     – Bad description: “Client slow to connect” 

     – Good description: “Client associations are rejected with Status17 severathey associate successfully.” 

    Troubleshooting 101

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    12/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Basics

    Step 2: Understand any possible triggers

     – If something previously worked but no longer works, there should be an trigger

     – Understanding any and all configuration or environmental changes coulda trigger

    Step 3: Know the expected behavior

     – If you know the order of expected behavior that is failing, defining where

    breaks down (Problem Description) is better than defining the end result. – Example: “One way audio between Phone A and B, because Phone A do

     ARP Response for Phone B” 

    Troubleshooting 101

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    13/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Basics

    Step 4: Reproducibility

     –  Any problem that has a known procedure to reproduce (or frequently ranshould be easy to diagnose

     – Being able to easily validate or disprove a potential solution saves time bto quickly move on to the next theory

     – If a problem is reproducible in other environments with a known proceducan facilitate internal testing and proposed fix/workaround verification

    Debugs and Captures of working scenarios can help pin point whethe difference is

    Troubleshooting 101

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    14/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Basics

    Wireless Sniffer

     – Example: Linksys USB600N with Omnipeek TAC can publish Omnipeek-RA if you have compatible HW

    Windows 7 with Netmon 3.4 https://supportforums.cisco.com/docs/DOC-16398 

    Mac OS X 10.6+ https://supportforums.cisco.com/docs/DOC-19212 

    Wired Packet Capture

     – Example: Wireshark Use for spanned switchports of AP/WLC or client side data

    Spectrum Analyzer

     – Spectrum Expert with Card or Clean-Air AP

    The “Client Debug”

     AP Packet Capture

    Recommended Tools

    https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-19212https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-16398https://supportforums.cisco.com/docs/DOC-16398

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    15/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Wireless LANs

    Software and Support

    Troubleshooting Basics

    AP Discovery/Join

    WLC Config/Monitoring

    Client Connectivity

    Mobility

    Packet Analysis

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    16/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    AP Discover/Join

     AP Runs Hunto Find Candid

    to

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    17/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    AP Discover/Join

     AP Discovery Request sent to knownand learned WLCs

    Broadcast

     – Reaches WLCs with MGMT Interfacein local subnet of AP

     – Use “ip helper -address ” with “ipforward-protocol udp 5246” 

    Dynamic – DNS: cisco-capwap-controller

     – DHCP: Option 43

    Configured (nvram)

     – High Availability WLCsPri/Sec/Ter/Backup

     – Last WLC

     –  All WLCs in same moblast WLC

     – Manual from AP - “capcontroller ip address

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    18/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    AP Discover/Join

    WLCs send Discovery Response back to AP

     – Name, Capacity, AP Count, Master?, AP-MGR, Load per AP-MGR

     AP selects the single best WLC candidate from

     – High Availability Config: Primary/Secondary/Tertiary/Backup

     – Master Controller

     – Greatest available capacity

     – Ratio of total capacity to available capacity

     AP sends single Join Request to best candidate

     – WLC responds with Join Response

     –  AP joins and receives config (or downloads image if not correct)

    Join Process

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    19/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    AP Discover/Join

    “Lightweight AP (LAP) Registration to a Wireless LAN Controller (W

    Document ID 70333  Make sure date/time on WLC is accurate (certificates)! 

    NAT

    Config network ap-discovery nat-ip-only

    From AP

    Debug ip udp

    Debug capwap client events

    From WLC

    Debug mac addr (Radio mac if running full k9w8 imag

    Debug capwap [event/error/packet] enable

    Troubleshooting AP Discover/Join

    http://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtmlhttp://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtmlhttp://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtmlhttp://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtml

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    20/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Wireless LANs

    Software and Support

    Troubleshooting Basics

     AP Discovery/Join

    WLC Config/Monitoring

    Client Connectivity

    Mobility

    Packet Analysis

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    21/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    WLC Supportability

     – Methods of Management

     – Using the GUI

     – Important Show Commands (CLI)

     – Important Debugs (CLI)

     – Best Practices

    Supportability - WLC

     AP Supportability

     ‒ Methods of Accessing the A

     ‒ Important Show Command

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    22/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Methods of Management 

    GUI – HTTPS (E) / HTTP (D)

    CLI

     – Console

     – SSH (E) / Telnet (D)

    SNMP – V1 (D) / V2 (E) – Change me!

     – V3 (E) – Change me

    Note: Management Via Wireless Clients (D)

    Supportability - WLC

    Default Mode

    (E)=Enabled (D)=

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    23/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Using the GUI 

    Monitor

     AP/Radio Statistics

    WLC Statistics

    Client Details

    Trap Log

    Supportability - WLC

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    24/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Using the GUI 

    Wireless > All APs

     AP list shows AP Physical UP Time

     APs are sorted by Controller Associated Time

    Check bottom of AP list for any recent AP disruptions

    Select AP to see Controller Associated Time (duration)

    WLC Config/MonitoringSupportability - WLC

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    25/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Using the GUI 

    Management

    SNMP Config

    Logs

    Tech Support

    Supportability - WLC

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    26/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Important Show Commands (CLI) 

    Show run-config

     –Must have! No exceptions!

     –“show run-config commands” (like IOS show running-config)

     –“show run-config no-ap” (no AP information added)

    Show tech-support

    CLI Tip

     –Log all output

     –Config Paging Disable

    Supportability - WLC

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    27/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Important Debugs (CLI) 

    Debug client

     –Client Involved? Must Have! No Exceptions

    Debug capwap enable

    CLI Tips

     –Log all output

     –Debugs are session based, they end when session ends

     –“Config session timeout 60”, sets 60 minute idle timeout 

     –Debug disable-all  (Disables all debugs)

    Supportability - WLC

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    28/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Best Practices

    Change default SNMP Parameters

    Configure Syslog for WLC and AP

    !!AP default behavior is to Broadcast syslog!!

    Enable Coredump for WLC and AP Configure NTP Server for Date/Time

    Supportability - WLC

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    29/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    AP Supportability

    Methods of Accessing the AP

     – Console

     – Telnet (D) / SSH (D)

     – No GUI support

     –  AP Remote Commands

    Enabling Telnet/SSH

     – WLC CLI: config ap [telnet/ssh] enable

     – WLC GUI: Wireless > All APs > Select AP > Advanced > Select [telnet/s

    Default Mode

    (E)=Enabled (D)=

    Supportability

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    30/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     AP Remote Commands (WLC CLI) 

    Debug AP enable

    Enables AP Remote Debug

     AP Must be associated to WLC

    Redirects AP Console output to WLC session

    Debug AP command “”  Output is redirected to WLC session

     AP runs IOS, numerous generic IOS commands available

    AP SupportabilitySupportability

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    31/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Show Commands (AP CLI or WLC Remote Cmd) 

    Show controller Do[0/1] (or Show Tech)

    Must have! Before/During/After event

    Show log

    WLC: show ap eventlog

    Show capwap client

    CLI Tips

    Debug capwap console cli

    Debug capwap client no-reload

    AP Supportability

    Supportability

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    32/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Supportability

     – WLC

     –  AP

    WLANs

    RRM / Radio / RF

    Wireless LAN Controller Config Analyzer (WLCCA)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    33/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

     AP “Default Group” consists of all WLANs ID 1-16 and cannot be m

     AP Groups must be created for WLAN ID 17+

     AP Groups override the Interface configured local to the WLAN

     AP Groups override default RF Profiles

    WLANs – AP Groups

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    34/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/MonitoringWLANs - Tweaks

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    35/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Supportability

     – WLC

     –  AP

    WLANs

    RRM / Radio / RF

    Wireless LAN Controller Config Analyzer (WLCCA)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    36/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RRM / Radio / RF

    There are generally two common scenarios or issues involving RR

     APs power change frequency (too much or not at all) – Nearby APs list meets the general rule of RSSI from 3rd closest AP is be

    Power Threshold

     – TPC Tuning may be required

     APs not changing channel

     – Check if other APs are in each others neighbor list

     –  Already established channel plan might not change APs without just cau

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    37/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RRM / Radio / RF

    show ap auto-rf [802.11a/b]

    Load Information – Receive Utilization.. 0 % Rx load to Radio

     – Transmit Utilization.. 2 % Tx load from Radio

     – Channel Utilization.. 12 % % Busy

     Nearby APs

     –  AP 00:16:9c:4b:c4:c0 slot 0.. -60 dBm on 11 (10.10.1.5)

     –  AP 00:26:cb:94:44:c0 slot 0.. -64 dBm on 11 (10.10.1.4)

    Show AP Auto-RF (In Run-Config)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    38/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RRM / Radio / RF

    Power Assignment Leader

    Power Threshold

    Consider Minimum Power Level Assignment

    Radio – TPC Tuning

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    39/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RRM / Radio / RF

    RF Profiles let you make the same TPC settings but for specific gro

    Radio – TPC Tuning – RF Profiles

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    40/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RRM / Radio / RF

    If channels change too frequently, DCA may need to be made lessrun at longer intervals

    DCA Tuning

    RRM / R di / RF

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    41/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RRM / Radio / RF

    In some large

    environments with new APs being deployed,STARTUP mode maybe beneficial

    Previously this requireda WLC REBOOT, but

    can be accomplished byRF Groupingconfiguration

    DCA – STARTUP Mode

    RRM / R di / RF

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    42/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RRM / Radio / RF

    Clean Air can give a remote viewinto the general RF environment

    around an AP

    RF – Clean Air

    WLC C fi /M it i

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    43/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    SE-Connect or Local Mode

    Obtain Spectrum Key

    Connect to Remote Sensor

    Spectrum Expert with Clean Air

    Spectrum Expert with Clean Air

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    44/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC C fi /M it i

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    45/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config/Monitoring

    Supportability

     – WLC

     –  AP

    WLANs

    RRM / Radio / RF

    Wireless LAN Controller Config Analyzer (WLCCA)

    WLC Config Analyzer (WLCCA)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    46/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config Analyzer (WLCCA)

    Main objective: Save time while analyzing configuration files from W

     Audit Checks

    Support Forums DOC-1373 

    WLC Config Analyzer (WLCCA)

    https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    47/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    WLC Config Analyzer (WLCCA)Support Forums DOC-1373 

    Secondary objective:

    Carry out RF analysis

    Troubleshooting Wireless LANs

    https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373https://supportforums.cisco.com/docs/DOC-1373

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    48/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Wireless LANs

    Software and Support

    Troubleshooting Basics AP Discovery/Join

    WLC Config/Monitoring

    Client Connectivity

    Mobility Packet Analysis

    Steps to Building an 802.11 Connection

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    49/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    1. Listen for Beacons

    2. Probe Request

    3. Probe Response4. Authentication Request

    5. Authentication Response

    6. Association Request

    7. Association Response

    8. (Optional: EAPOL Authentication)

    9. (Optional: Encrypt Data)

    10. Move User Data

    State 1:

    Unauthenticated,

    Unassociated

    State 2:

    Authenticated,

    Unassociated

    State 3:

    Authenticated,

    Associated

    802.11

    A

    Understanding the Client State

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    50/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Understanding the Client StateName Description

    8021X_REQD 802.1x (L2) Authentication Pending

    DHCP_REQD IP Learning State

    WEBAUTH_REQD Web (L3) Authentication Pending

    RUN Client Traffic Forwarding

    (Cisco Controller) >show client detail 00:16:ea:b2:04:36

    Client MAC Address............................... 00:16:ea:b2:04

    ….. 

    Policy Manager State............................. WEBAUTH_R

    00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

    The Client Debug

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    51/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     A multi-debug macro – (Cisco Controller) >debug client 00:16:EA:B2:04:36

     – (Cisco Controller) >show debug

     – MAC address ................................ 00:16:ea:b2:04:36

     – Debug Flags Enabled: 

    dhcp packet enabled

    dot11 mobile enabled

    dot11 state enabled

    dot1x events enabled

    dot1x states enabledpem events enabled

    pem state enabled

    CCKM client debug enabled

    The Client Debug

    The Client Debug

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    52/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    The Client Debug• 3 Simultaneous MAC Addresses in 7.2

    • Up to 10 Simultaneous MAC Addresses in 7.3 and later

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    53/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    g g

     Association (Start)

    L2 Authentication (8021X_REQD) Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc Tips and Tricks

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    54/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    The Client Debug Walkthrough

    Association (Start)

    L2 Authentication (8021X_REQD) Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc Tips and Tricks

    Association

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    55/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    (Cisco Controller) >debug client 00:16:EA:B2:04:36 

    (Cisco Controller) >

    (Cisco Controller) >

    Association received from mobile on AP 00:26:cb:94:44:c0

    0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:162 Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface

     Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3‘ 

    STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0

    Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36

    0.0.0.0 START (0) Initializing policy

    0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)

    0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client

    0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1

    apfMsAssoStateInc

    apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Ass

    Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds

    Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

    Association

    Association

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    56/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Association

     Association received

     Association Request, client did not “Roam” (Reassociate) 

     AP Base Radio = 00:26:cb:94:44:c0

    vapId 1, site 'default-group', interface '3‘

    vapId = WLAN # (Wlan 1)

    site = AP Group (default-group)

    Interface = Dynamic Interface name (3)

    vlan 3

    Vlan = Vlan # of Dynamic Interface

    Association received from mobile on AP 00:26:cb:94:44:c0

    0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:162

     Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interfa

     Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3'

    Association

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    57/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Association

    STA - ratesMandatory Rates (>128) = (#-128)/2

    Supported Rates (

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    58/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Association

    0.0.0.0 START

    0.0.0.0 = IP we know for client (In this case nothing)

    Change state to 8021X_REQDPassed association, moving client to next state: 8021X_REQD

    Scheduling deletion

    Session Time on WLAN (1800 seconds in this case)

    0.0.0.0 START (0) Initializing policy

    0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)

    0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)

    0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client

    0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1

    apfMsAssoStateInc

    apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Ass

    Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds

    Association

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    59/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Common Assoc Response Failures:

    1 – Unknown Reason – Anything not matching defined reason codes

    12 – Unknown or Disabled SSID

    17 – AP cannot handle any more associations (Load Balancing)

    18 – Client is using a datarate that is not allowed

    35 – WLAN requires the use of WMM and client does not support it

    201 – Voice client attempting to connect to a non-platinum WLAN

    202 – Not enough available bandwidth to handle a new voice call (CAC Reje

    Association

    Slot 0 = B/G(2.4) Radio

    Slot 1 = A(5) Radio

    Sending Assoc Response Status 0 = Success

     Anything other than Status 0 is Failure

    Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

    Association - Takeaway

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    60/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    y

     Association vs. Reassociation

    Debug shows AP, Slot, AP-Group, WLAN ID, Interface, Data Rates, Encryption type

     Association Response

    Confirms if Client is associated

    Defines reason if denied

    Further troubleshootingMay require Wireless Sniffer or capture at AP Switchport

    If not sending Assoc Request, must know why from Client

    Trying disabling WLAN features to “dumb it down” 

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    61/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    g g

     Association (Start)

    L2 Authentication (8021X_REQD)

    Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc

    Tips and Tricks

    802.1X Authentication

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    62/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    EAP-ID-Request

    Rest of the EAP Conversation

    Radius-Access

    (Key)

    EAP-Success

    EAPOL-START

    EAP-ID-ResponseRADIUS (EAP-ID_R

    Supplicant Authenticator

    The Supplicant Derives the

    Session Key from User Password or

    Certificate and Authentication ExchangeSe

    802.1X Authentication Association + 802 1x

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    63/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     Association + 802.1x

    Probe Request

    Probe Response

     Auth Request

     Auth Response

     Association Request

     Association Response

    EAP Start

    EAP ID Request

    EAP ID Response

    EAP Method

    EAP Success

    EAPoL 4 way Exchange

    DATA

     AP W

    Between 4 and

    20+ frames

    WPA2-AES-802.1X

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    64/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

    Station 00:16:ea:b2:04:36 setting dot1x reauth timeout = 1800

    dot1x - moving mobile 00:16:ea:b2:04:36 into Connecting state

    Sending EAP-Request/Identity to mobile 00:16:ea:b2:04:36 (EAP Id 1)

    Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36

    Username entry (cisco) created for mobileReceived Identity Response (count=1) from mobile 00:16:ea:b2:04:36

    EAP State update from Connecting to Authenticating for mobile 00:16:ea:b2:04:36

    dot1x - moving mobile 00:16:ea:b2:04:36 into Authenticating state

    ………………….. 

    Entering Backend Auth Req state (id=3) for mobile 00:16:ea:b2:04:36

    Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3)

    Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36

    Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25)

    ...........................

    Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 10, EAP Type 25)

    Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36

    Processing Access-Challenge for mobile 00:16:ea:b2:04:36Entering Backend Auth Req state (id=11) for mobile 00:16:ea:b2:04:36

    Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 11)

    Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36

    Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 11, EAP Type 25)

    Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36

    Processing Access-Accept for mobile 00:16:ea:b2:04:36 

    ***OR***Processing Access-Reject for mobile 00:16:ea:b2:04:36

    Common EAP Types

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    65/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    1 – Identity

    2 – Notification

    3 – NAK

    4 – MD5

    5 – OTP

    6 – Generic Token

    13 – EAP TLS

    17 – LEAP

    18 – EAP SIM

    21 – EAP TTLS

    25 – PEAP

    43 – EAP-FAST

    Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3

    Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EA

    WPA(2) - PSK Authentication

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    66/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Probe Request

    Probe Response

     Auth Request

     Auth Response

     Association Request

     Association Response

    EAPoL 4 way Exchange

    DATA

     AP W

    WPA(2) – PSK Authentication (cont.)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    67/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

    Creating a PKC PMKID Cache entry for station 00:16:ea:b2:04:36 (RSN 2)

     Adding BSSID 00:26:cb:94:44:c0 to PMKID cache for station 00:16:ea:b2:04:36

    New PMKID: (16)

    [0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd

    Initiating RSN PSK to mobile 00:16:ea:b2:04:36

    dot1x - moving mobile 00:16:ea:b2:04:36 into Force Auth stateSkipping EAP-Success to mobile 00:16:ea:b2:04:36

    Including PMKID in M1 (16) 

    [0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd

    Starting key exchange to mobile 00:16:ea:b2:04:36, data packets will be dropped

    Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36

    state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00

    Received EAPOL-Key from mobile 00:16:ea:b2:04:36

    Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36

    Received EAPOL-key in PTK_START state (message 2) from mobile 00:16:ea:b2:04:36

    Stopping retransmission timer for mobile 00:16:ea:b2:04:36Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36

    state PTKINITNEGOTIATING (message 3), replay counter 00.00.00.00.00.00.00.01

    Received EAPOL-Key from mobile 00:16:ea:b2:04:36

    Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36

    Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile 00:16:ea:b2:04:36

    apfMs1xStateInc

    0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4)  last state L2AUTHCOMPLETE (4) 

    WPA2- PSK - Failed

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    68/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Starting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be dropped

    Sending EAPOL-Key Message to mobile 00:1e:8c:0f:a4:57

    state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00

    Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57

    Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57

    Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57

    Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57

    802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57

    Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57

    Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57

    Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57

    Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57

    Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57

    802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57

    Retransmit 2 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57

    ………………… 802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57

    Retransmit failure for EAPOL-Key M1 to mobile 00:1e:8c:0f:a4:57,

    retransmit count 3, mscb deauth count 3

    Blacklisting (if enabled) mobile 00:1e:8c:0f:a4:57

    apfBlacklistMobileStationEntry2 (apf_ms.c:4192) Changing state for mobile 00:1e:8c:0f:a4:57 on

     AP 00:16:9c:4b:c4:c0 from Associated to Exclusion-list (1)

    L2 Authentication - Takeaway

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    69/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    8021X_REQD means L2 Authentication pending

     Authentication/Encryption has not be established

    PSK is 802.1X, key is derived from PSK not AAA

    If “Processing Access-Reject” 

     AAA/RADIUS Rejected the user (not the WLC)

    If “Processing Access- Accept” 

     AAA/Radius Accepted the userM1-M4 should follow

    Further Troubleshooting

    Debug aaa [all/event/detail/packet] enable

    Debug dot1x [aaa/packet] enable

    802.1X Authentication Roaming

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    70/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Probe Request

    Probe Response

     Auth Request

     Auth Response

    Reassociation Request

    Reassociation Response

    EAP Start

    EAP ID Request

    EAP ID Response

    EAP Method

    EAP Success

    EAPoL 4 way Exchange

    DATA

     AP2 W

    Between 12 and20+ packets

    DATA AP1

    802.1X Authentication Roaming

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    71/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    802.1x + WPA2 FSR (PMKID Caching) is like PSK

    Probe Request

    Probe Response

     Auth Request

     Auth Response

    Reassociation Request

    Reassociation Response

    EAPoL 4 way Exchange

     AP2 W

    DATA

     AP1

    6 packets

    DATA

    802.1X with CCKM Authentication Roaming

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    72/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    CCKM (WPA1-TKIP or WPA2-AES)

    Probe Request

    Probe Response

     Auth Request

     Auth Response

    Reassociation Request

    Reassociation Response

     AP2 W

    DATA

     AP1

    2 packets

    DATA

    Association - FSR

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    73/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Processing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36

    CCKM: Mobile is using CCKM

    CCKM: Processing REASSOC REQ IE

    Including CCKM Response IE (length 62) in Assoc Resp to mobile

    Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) Vap Id 6 Slot

    FSR

    CCKM - WPA

    CCKM - WPA2

    WPA2 PKC

    WPA2 "Sticky"

    OR

    Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36

    Received RSN IE with 1 PMKIDs from mobile 00:16:ea:b2:04:36

    Received PMKID: (16)

    [0000] cb bc 27 82 88 14 92 fd 3b 88 de 6a eb 49 be c8

    Found an entry in the global PMK cache for station

    Computed a valid PMKID from global PMK cache for mobile

    * WPA2 “Sticky” PMKID Caching is now supported in 7.2 WLC Release with limited scale

    This at least allows some form of Fast Secure Roaming for “Sticky” clients (like Apple).

    802.11r Roaming

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    74/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    AP1

     

    Client

      ProbReq 

      ProbResq 

    FT req via 802.11 auth/A ctio n

    frame 

    FT resq v ia 802.11 auth / 

    Act ion frame 

    AssocReq wi th QOS req 

    AssocResp wi th QOS req 

    AP2

    DATAtransfer

    via AP1

    DATA

    transfe

    via AP

    ROAMIN

    WPA2 - .11r Client (Fast Transition)

    802.11r Over the Air Roaming

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    75/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     AP1

    Client

    Roaming direction

     A  s  s  o  c  i   a  t   e  d    w  i   t   h   

    o  l   d     A  P   

     AP2, 3, 4

     8 0 2. 1

     1   F  T  a

     u  t  h  r e q 

     8 0 2. 1

     1   F  T  a

     u  t  h  r e s p

      R e a s s

     o c  i a  t  i

     o n   R e

     q 

      R e a s s

     o c  i a  t  i

     o n   R e

     s p

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    76/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     Association (Start)

    L2 Authentication (8021X_REQD)

    Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc

    Tips and Tricks

    Client DHCP

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    77/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state00:16:ea:b2:04:36 apfMs1xStateInc

    00:16:ea:b2:04:36 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4)

    00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 3 apVap

    00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vap00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7)

    00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4755, Adding TMP rule00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

    00:16:ea:b2:04:36 Stopping retransmission timer for mobile 00:16:ea:b2:04:36

    *pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

    ...................

    00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 29, encap 0xec03)

    ...................

    00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)

    ...................00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

    00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0

    Client DHCP

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    78/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Client is in DHCP_REQD state

    Proxy Enabled:

    DHCP Relay/Proxy

    Between WLC and Server

    Required for Internal DHCP

    Proxy Disabled:

    Between Client and Server

    DHCP is broadcast out VLANIP helper or other means required

    Client State = “DHCP_REQD

    DHCP Proxy Enabled

    Client DHCP Discover

    Unicast to DHCP Servers

    DHCP Offer from Server

    DHCP ACK from Server

    IP Address Learned

    Client DHCP Request

    DHC

    Client

    B

    DHCP Proxy Enabled – DHCP Discover

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    79/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    *pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

    32.151: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0x

    32.151: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings:

    dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,dhcpGateway: 0.0.0.0, dhcpRelay: 0.0.0.0 VLAN: 0

    32.151: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.1

    (local address 10.10.1.4, gateway 10.10.1.1, VLAN 0, port 29)

    32.151: 00:16:ea:b2:04:36 DHCP transmitting DHCP DISCOVER (1)

    32.151: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 1

    32.151: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0

    32.152: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36

    32.152: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0

    32.152: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.4

    32.152: 00:16:ea:b2:04:36 DHCP requested ip: 10.99.76.147

    32.152: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.1 (len 346, port 29, vlan 0)

    32.152: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings:

    dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,

    dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 0

    32.152: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE

    DHCP Proxy Disabled – DHCP Discover

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    80/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    *pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

    *00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)

    *00:16:ea:b2:04:36 DHCP processing DHCP DISCOVER (1)

    *00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0

    *00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0

    *00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36

    *00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0

    *00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0

    *00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86

    *00:16:ea:b2:04:36 DHCP successfully bridged packet to DS

    Learning IP without DHCP

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    81/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Client IP can be learned by ways other than DHCP

    Client sends gratuitous ARP or ARP Request (Static Client)

    Client sends IP packet (Orphan Packet), we learn IP

    DS sends packet to client, we learn IP from DS

    Seen with mobile devices that talk before validating DHCP

    Up to client to realize their address is not valid for the subnet

    DHCP Required on WLAN for preventing this

    *Orphan Packet from 10.99.76.147 on mobile

    *0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

    *Installing Orphan Pkt IP address 10.99.76.147 for station

    *10.99.76.147 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

    Client DHCP - Takeaway

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    82/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    DHCP_REQD means Learning IP State

    Only “Required” if enabled on the WLAN 

    If Proxy is enabledConfirm DHCP Server on Interface (or Wlan) is correct

    DHCP Server may not respond to WLC Proxy (Firewalls?)

    If Proxy is disabled, DHCP is similar to wired client

    Further Troubleshooting

    Check DHCP Server for what it believes is happeningIf WLC does not show a BOOTREQUEST, confirm the client request arrive

    and leaves in the configured way

    If still believed to be on WLC: debug dhcp message enable

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    83/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     Association (Start)

    L2 Authentication (8021X_REQD)

    Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc

    Tips and Tricks

    Webauth*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    84/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    *pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xe

    ……………………………...  

    *DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to WEB

    last state WEBAUTH_REQD (8)*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) pemAdvanceState2 5170, Adding TMP r*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Successfully plumbed mobile rule (ACL I

    *DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile

    *pemReceiveTask: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 2, dtlFlags 0x0*pemReceiveTask: 00:16:ea:b2:04:36 Sent an XID frame

    *apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile

    *apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile

    *apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile……………………………… 

    *emWeb: 00:16:ea:b2:04:36 Username entry (cisco) created for mobile*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14) last state WEBAU

    *emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last st*emWeb: 00:16:ea:b2:04:36 Session Timeout is 1800 - starting session timer for the mobile

    *emWeb: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063

    *emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, Toke

    3, IPv6 intf id = 8

    *emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Successfully plumbed mobile rule (ACL ID 255)

    *pemReceiveTask: May 17 22:25:16.578: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1,

    Webauth RedirectWebauth

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    85/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Client in WEBAUTH_REQD state

     ARP and DNS must be functional

    Client attempts to browse internet

    WLC “Hijacks” the handshake 

    Client redirects to Virtual Interface

    Certificate negotiation if applicable

    Webauth page is displayed

    Client authenticates

    Client State

    “WEBAUTH_RE

     ARP and DNS Fu

    3-Way Handshake

    HTTP GET

    200 Respons

    3-Way Handsh

    HTTP(S) GE

    Successful Authen

    Client State = “R

    Webauth Page Dis

    Confirm ARP and DNS Function

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    86/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     AR

    Capture from Wireless Adapter

    Webauth Redirect

    WLC Re

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    87/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Webauth Redirect WLC ReS

    WLC Re

    S

     Address for Cl

    Redirect to (V

    IP/Name

    Redir

    Interfac

    Clie

    If WEBAUTH REQD, then not authenticated

    Webauth - Takeaway

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    88/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    If WEBAUTH_REQD, then not authenticated

    Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv6*(7.0 and ear

    If not redirected, can client browse to virtual IP?

    Cert issue? Consider disabling HTTPS for HTTP webauth

    Most common scenario involves ARP/DNS failure

    Must confirm that client actually sends TCP SYN (http) to IP

    If proven that TCP SYN is sent and WLC does not SYN ACK, then

    be a WLC side problemdebug client debug webauth enable

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    89/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     Association (Start)

    L2 Authentication (8021X_REQD)

    Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc

    Tips and Tricks

    Run State

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    90/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    RUN State is the Client Traffic Forwarding State

    Client is Connected and should be functional

    10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

    10.10.3.82 RUN (20) Reached PLUMBFASTPATH: from line 5273

    10.10.3.82 Added NPU entry of type 1, dtlFlags 0x0

    OR

    10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14)

    10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20)

    Session Timeout is 1800 - starting session timer for the mobile

    10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063

    10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    91/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     Association (Start)

    L2 Authentication (8021X_REQD)

    Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc

    Tips and Tricks

    Deauthenticated Client Idle Timeout

    O ft t ffi i d f Cli t t AP

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    92/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Occurs after no traffic received from Client at AP

    Default Duration is 300 seconds

    Session Timeout

    Occurs at scheduled duration (default 1800 seconds)

    Received Idle-Timeout from AP 00:26:cb:94:44:c0, slot 0 for STA 00:1e:8c:0f:a4:57

    apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds

    apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!

    Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

    apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on

     AP 00:26:cb:94:44:c0 from Associated to Disassociated

    Scheduling deletion of Mobile Station: (callerId: 45) in 10 seconds

    apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!

    Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

    Deauthenticated Client WLAN Change

    f

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    93/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Modifying a WLAN in anyway Disables and Re-enables WLAN

    apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile

    00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated

    Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

    apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1Scheduling deletion of Mobile Station: (callerId: 30) in 1 seconds

    apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!

    apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on

     AP 00:26:cb:94:44:c0 from Associated to Disassociated

    Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

    Manual Deauth

    From GUI: Remove Client

    From CLI: config client deauthenticate

    Deauthenticated Client

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    94/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Retransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3, mscb decount 0

    Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)

     Authentication Timeout

     Auth or Key Exchange max-retransmissions reached

    Cleaning up state for STA 00:1e:8c:0f:a4:57 due to event for AP 00:26:cb:94:44:c0(0)

    apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile

    00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated

    Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)

     AP Radio Reset (Power/Channel)

     AP disasassociates clients but WLC does not delete entry

    Deauthentication - Takeaway

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    95/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Client can be removed for numerous reasons

    WLAN change, AP change, configured interval

    Start with Client Debug to see if there is a reason for a client’s dea

    Further Troubleshooting

    Client debug should give some indication of what kind of deauth is happen

    Packet capture or client logs may be require to see exact reason

    The Client Debug - Walkthrough

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    96/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     Association (Start)

    L2 Authentication (8021X_REQD)

    Client Address Learning (DHCP_REQD)

    L3 Authentication (WEBAUTH_REQD)

    Client Fully Connected (RUN)

    Deauth/Disassoc

    Tips and Tricks

    Tips and Tricks

    C ll li d b f d d d i

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    97/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Collect a client debug for an extended duration

    Several roams, deauths, failures, etc… 

    Use an enhanced text editor with filter or “find all” I use Notepad++

    Find All

    “Association Received” (will also pull reassociations) 

    “Assoc Resp” 

    “Access-Reject” 

    “timeoutEvt” 

    Tips and Tricks

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    98/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Tips and Tricks

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    99/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Troubleshooting Wireless LANs

    S ft d S t

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    100/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Software and Support

    Troubleshooting Basics

     AP Discovery/Join

    WLC Config/Monitoring

    Client Connectivity

    Mobility

    Packet Analysis

    Mobility—Intra-ControllerClient Roams Between Two APs on the Same Controller

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    101/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Mobility—Inter-Controller (Layer 2)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    102/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Mobility—Layer 3

    Layer 3 roaming (a.k.a. anchor/foreign)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    103/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    New WLC does not have an interface on the subnet the client is on

    New WLC will tell the old WLC to forward all client traffic to the new WLC

     Asymmetric traffic pathestablished

    (deprecated)

    Symmetric traffic path

     –  

    Mobility— L2 Inter WLC

    OldController

    Client

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    104/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    New

    Controller

    Controller

    3. mmMobileAnnounce

    1.Association Req.

    4. mmMobileHandoff

    2.Association Resp.

    Local

    DATA

    DATA

    Debug Client

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    105/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    MobileAnnounce

    MobileHandoff

    Mobility— L3 Inter WLC

    OldController

    Client

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    106/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    New

    Controller

    3. mmMobileAnnounce

    1.Association Req.

    4. mmMobileHandoff

    2.Association Resp.

    Foreign

    DATA

    DATA

    Anchor

    (EOIP) DATA

    Mobility— L3 Inter WLC Debug Client

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    107/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    MobileAnnounce

    MobileHandoff

    Mobility— L3 Inter WLC Debug Client

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    108/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Anchor

    Mobility Group vs. Mobility Domain

    Mobility Group - WLCs with the same group name

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    109/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Mobility Group WLCs with the same group name

    L2/L3 Handoff

     Auto Anchoring

    Fast Secure Roaming

     APs get all of these as a Discover candidate

    Mobility Domain - WLCs in the mobility list

    L2/L3 Handoff

     Auto Anchoring

    Sent between all WLCs, by member with lowest MAC

     – Control Path = UDP 16666 (30 Seconds)

    Mobility Data/Control Path

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    110/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

     – Data Path = EoIP Protocol 97 (10 Seconds)

     – debug mobility keep-alive enable

    Troubleshooting Wireless LANs

    Software and Support

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    111/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    pp

    Troubleshooting Basics

     AP Discovery/Join

    WLC Config/Monitoring

    Client Connectivity

    Mobility

    Packet Analysis

    Wireshark Tutorial

    Default Wireshark view might look like this:

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    112/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Wireshark Tutorial

    Newer versions of Wireshark have a feature for “Apply as Column”

    This will take any decodable parameter and make a column

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    113/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    y p

    Wireshark Tutorial Within seconds your wireshark can also have:

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    114/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Wireshark Tutorial

    Filtering data is just as easy

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    115/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Wireshark Tutorial - CAPWAP

    User data is encapsulated in CAPWAP

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    116/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Wireshark Tutorial Wireshark can also de-encapsulate CAPWAP DATA

    Edit > Preference > Protocols > CAPWAP

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    117/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Wireshark Tutorial

    With CAPWAP de-encapsulated you can see all the packets to/from(b t AP d WLC)

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    118/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    (between AP and WLC)

    Sniffer Mode AP

    Select channel to Sniff

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    119/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Select destination for traffic

    Sniffer Mode AP

    Omnipeek has a Remote Adapter to capture this data

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    120/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Wireshark, just capture network adapter

    NOTE: Wireshark does not open the port UDP 5000PC will send ICMP Unreachables

    Sniffer Mode AP

    With wireshark, filter !icmp.type == 3

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    121/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Data (UDP 5000) still not intelligible yet

     – Decode as Airopeek (Peekremote in wireshark 1.8+)

    Sniffer Mode AP

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    122/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    AP Packet Dump

    In 7.3 WLC release, we added an AP packet dump feature that canpackets from a wireless client at the AP radio.

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    123/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    p

    Much easier than performing an Over-The-Air capture, can be perf

    remote locations The APs will send the packet dump to the configured FTP server

    AP Packet Dump – FTP Server Required

    Feature requires use of a standard FTP server running on a netwoworkstation, or laptop i.e. IIS, Filezilla, WS FTP, 3CD, etc.

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    124/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    , p p , , , ,

    FTP server needs to be accessible by the APs capturing packets n

    controller Multiple simultaneous file upload connections will be initiated to the

    —One for the AP designated in the start command

    —One for each AP that is an RF neighbor of the AP desigthe start command – on th e same contro l ler only

    File name format example: 3602-15508-223042013 _ 160038.pcap AP Name

    Controller NameDate ddmmyyyy

    Time hhmmsec

    AP Packet Dump Commands

    config ap packet-dump ftp serverip path usernampassword

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    125/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    (Cisco Controller) >show ap packet-dump status

    Packet Capture Status............................ StoppedFTP Server IP Address............................ 172.16.0.11FTP Server Path.................................. \FTP Server Username.............................. ciscoapFTP Server Password.............................. ********Buffer Size for Capture.......................... 4096 KBPacket Capture Time.............................. 10 MinutesPacket Truncate Length........................... Unspecified

    Packet Capture Classifier........................ 802.11 Data

    AP Packet Dump Filters

    • First define packets to be captured by enabling specific classifiers CLI

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    126/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    — config ap packet-dump classifier enable/disable

    — Only the following pre defined classifiers are available• arp• broadcast• control• data• dot1x• iapp• Ip• management• multicast

    • tcp• udp

    • Classifiers are enabled one at a time - more than one classifier can be time

    Starting the Packet Dump• Start the dump process from the controller CLI using

     – config ap packet-dump start

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    127/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    • Packet dump ends either when the capture timer expires or the promanually stopped from the controller CLI using

     –config ap packet-dump stop

    (Cisco Controller) >config ap packet-dump start 00:24:d7:45:4e:6c 3602-

    Client Mac Address............................... 00:24:d7:45:4e:6c

    FTP Server IP.................................... 172.16.0.11

    FTP Server Path.................................. \

    FTP Server Username.............................. ciscoap

    Buffer Size for Capture.......................... 4096 KB

    Packet Capture Time.............................. 10 Minutes

    Packet Truncate Length........................... Unspecified

    Packet Capture Classifier........................ 802.11 Data

     Are you sure you want to start capture ? (y/N)

    Files are not created until

    you answer yes here

    AP Packet Dump - dot1xThe 802.11 authentication & asso

    The dot1x process begi

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    128/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    The dot1x process begi

    The dot1x proc

    The remaining encrypted

    packets provide little useful

    information

    AP Packet Dump – Open/Webauth

    The 802.11 authentication & asso

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    129/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    The DHCP

    Process

    Details

     Available

    Summary - Key Takeaways

     Accurate Problem Description is crucial

    Understand the flow for a successful client connection, determine w

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    130/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    ,failing

    Know the tools that are available – Debugs, show commands

     – Packet captures – sniffer mode, AP packet dump

     – WLCCA for configuration analysis

     A few commands can go a long way

     – show run-config

     – debug client xx:xx:xx:xx:xx:xx

    Complete Your Online Session Evaluation

    Give us your feedback andyou could win fabulous prizes.Wi d d il

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    131/132

    © 2013 Cisco and/or its affiliates. All rights reserved.BRKEWN-3011 Cisco Public

    Maximize your Cisco Live exp

    free Cisco Live 365 account. DPDFs, view sessions on-dema

    live activities throughout the y

    Cisco Live 365 button in your

    log in.

    Winners announced daily.

    Receive 20 Cisco Daily Challengepoints for each session evaluationyou complete.

    Complete your session evaluationonline now through either the mobileapp or internet kiosk stations.

  • 8/19/2019 BRKEWN-3011 - Troubleshooting Wireless LANs (2013 Orlando) - 2 Hours

    132/132


Recommended