IOS-XE Troubleshooting · ESP FECP QFP Crypto Assist. interconn. PPE BQS ESP FECP Crypto Assist....

IOS-XE Troubleshooting Hands-on Lab

Olivier Pelerin, Technical Leader

Michal Stanczyk, Customer Support Engineer

Wen Zhang, Technical Leader

LTRARC-3500

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#LTRARC-3500

• Introduction to IOS-XE Platform Software/Hardware Architecture

• Day in the Life of a Packet

• Troubleshooting strategy and Tools

• Resource Consumption Monitoring

• Understanding and Extracting Platform Logs

• Embedded Packet Capture

• Data Plane Packet Tracing

• Advanced Data Pane Debugging

• Hands-on Lab exercise

• Wrapping up...

Agenda


Session Objectives

• To understand the IOS-XE (ASR1k, ISR4k, CSR1Kv) platform architecture

• Software

• Hardware

• Feature implementations

• Understand how features process packets through IOS-XE

• To demonstrate a systematic troubleshooting strategy

• To showcase various troubleshooting Tools and Capabilities

• To provide a hands-on experience on how to effectively troubleshoot the platform using these tools

5LTRARC-3500


Related Sessions

• BRKCRS-3147 - Advanced troubleshooting of the ASR1K and ISR (IOS-XE) made easy

• Olivier Pelerin – Technical Leader, Services

• Frederic Detienne – Distinguished Engineer, Services

• LABRST-2400 - Packet Capturing Tools in Routing Environments WISP Lab

6LTRARC-3500

ASR Series Hardware Architecture


ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

ESP

FECP

Crypto

Assist.

interconn.

PPE BQS

ASR1K Building Blocks

FECP

QFP

interconn.

RP

CPU

interconn. GE switch

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

RP

CPU


Midplane

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn. SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

Activ

e

Activ

e

Stb

y

Stb

y

Route Processor

Handles control plane traffic

Manages system

Embedded Service Processor

Handles forwarding plane traffic

SPA Interface Processor

Houses SPA’s

Queues packets in & out (FIFO)

8LTRARC-3500


ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

System Architecture Control Plane

RP

CPU


SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

RP

CPU


Midplane

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn. SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

Activ

e

Stb

y

Stb

y

Ethernet Out of Band Channel

(aka EOBC)

1Gbps Ethernet bus

Used by RP to program system

Used by system to notify RP

Inter Integrated Circuit (I2C) Bus

Slow (few kbps)

Used for system monitoring

(temp., OIR, fan speed,…)

EOBC switch in RP

SPA Control Link

Works between the SPA’s and SIP

Activ

e

9LTRARC-3500


ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

System Architecture Forwarding Plane

RP

CPU


SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

RP

CPU


Midplane

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn. SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

Activ

e

Activ

e

Stb

y

Stb

y

Hypertransport

10 Gbps Ethernet

Embedded Service Interconnect

aka ESI Bus

11.2 – 200 Gbps Forwarding Bus

Centralized Architecture

All traffic flows through ESP

10LTRARC-3500


Route Processor ArchitectureHighly Scalable Control Plane Processor

ESPs

2.5’’

Hard disk

Output clocks

SIPs ESPs RP SIPs RPESPsMiscCtrl

SIPs SIPs

Inputclocks

RP

ESI, 11.2-40 Gbps

SPA-SPI, 11.2Gbps

Hypertransport, 10Gbps

Other

GE, 1Gbps

I2C

SPA Control

SPA Bus

CPU(1.5 – 2.66 GHz Dual-core)

I2C Chassis

Management Bus ESI

Interconnect

EOBC

Gig Eth Switch

CPU

Memory

Mgmt

EthernetUSB

Console

& Aux

Route Processor

Manages all chassis functions

Runs IOSNot a traffic interface!

Management only

IOS Memory: RIB, FIB &

other processes

Determines BGP routing

table size

RP1: 4GB

RP2: 8&16GB

System

Logging

Core Dumps

Runs IOS, Linux OS

Manages boards and chassis

NVRAM

Bootdisk

Stratum-3 Network

clock circuit

33MB

RP1: 1GB

RP2: 2GB

Card Infrastructure

BITS

(input & output)

RP

CPU


11LTRARC-3500


Crypto(Nitrox-II CN2430)

FECP

ESP-xx Block Diagram

GE, 1Gbps

I2C

SPA Control

SPA Bus

ESI, 11.2Gbps

SPA-SPI, 11.2Gbps


Other

RPs RPs RPsESP SIPs

E-RP*PCI*

E-CSR

QFP

TCAM(10Mbit)

Resource DRAM(512MB)

Packet Buffer

DRAM(128MB)

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl

Packet Processor Engine

…

PPE1 PPE2 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPE40

BQS

Reset / Pwr Ctrl

Interconnect

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

Interconnect

12LTRARC-3500

ISR Series Hardware Architecture


ISR 4451-X Hardware Diagram

Data Plane(10 core)


PPE6 PPE7 PPE8 PPE10PPE9

Control Plane(4 cores)

Ctrl SVC1

SVC2 SVC3

FPGE

DRAM

Multi Gigabit

Fabric

DSP

SM-X

System

FPGA

Peripheral

Interconnect

DRAM

Console / Aux

Mgmt Ethernet

Flash

USB

4xPCIeDDR3 4xSGMI

DDR3

1xSGMI

10 Gbps/slot

NIMNIM

NIM

2Gb/slot

SM-X

10 Gbps XAUI

14LTRARC-3500


ISR 4451-X Hardware Diagram

Data Plane(10 core)


PPE6 PPE7 PPE8 PPE10PPE9

Control Plane(4 cores)

Ctrl SVC1

SVC2 SVC3

FPGE

DRAM

Multi Gigabit

Fabric

DSP

SM-X

System

FPGA

Peripheral

Interconnect

DRAM

Console / Aux

Mgmt Ethernet

Flash

USB

4xPCIeDDR3 4xSGMI

DDR3

1xSGMI

10 Gbps/slot

NIMNIM

NIM

2Gb/slot

SM-X

10 Gbps XAUI

1 Control Plane Core

RP and FECP-like roles

3 Services Core

10 Cores, 1 thread / core

5 fwd cores by default

4 remaining cores license

activated

Inline Cryptography

No Crypto Assist chip

Crypto “locks” core

True run-to-completion

No hardware TCAM

BQS on a core

One Core dedicated to BQS

Always active

(5+1 or 9+1 cores)

15LTRARC-3500


4351 Hardware Diagram (aka Utah)

Rangeley CPU


PPE6 PPE7 PPE8

GE Switch

PCIe Switch

DRAM

Front Panel Ethernet

NIM Slots x 2

Front Panel EthernetFront Panel Ethernet

SPI Flash

USB Host Ports

eMMc

USB-to-SD

System Glue Logic

FPGA

mSATA

(MO-300)

Mgmt Ethernet

NIM Slots x 2NGSM Slots x 2

NGSM slots x 2

Console, Aux & USB

Console

I2C to Modules

16LTRARC-3500


4351 Hardware Diagram (aka Utah)

Rangeley CPU


PPE6 PPE7 PPE8

GE Switch

PCIe Switch

DRAM

Front Panel Ethernet

NIM Slots x 2

Front Panel EthernetFront Panel Ethernet

SPI Flash

USB Host Ports

eMMc

USB-to-SD

System Glue Logic

FPGA

mSATA

(MO-300)

Mgmt Ethernet

NIM Slots x 2NGSM Slots x 2

NGSM slots x 2

Console, Aux & USB

Console

I2C to Modules

8 Cores @ 2.4 Ghz / 1 thread per core

1 core for RP/IOSd

1 core acting for Crypto & QoS

4 cores @ 1 thread/core for features

2 service cores

1 core as Crypto and BQS

1 core as RP hosting IOSd

2 service cores

2 cores QFP

2 cores QFP license activated

4331 and 4321 are similar; just less cores

and expansion slots

17LTRARC-3500


Crypto

FECP

Generic ESP Block Diagram

GE, 1Gbps

I2C

SPA Control

SPA Bus

ESI, 11.2Gbps

SPA-SPI, 11.2Gbps


Other

RPs RPs RPsESP SIPs

QFP Complex

TCAM Resource DRAMPacket Buffer

DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…


PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

Interconnect

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

18LTRARC-3500


Acronyms

• RP – Route Processor

• FP – Forwarding Processor = ESP (Embedded Service Processor)

• CPP – Cisco Packet Processor Compex= QFP (Quantum Flow Processor)

• PPE – Packet Processing Engine

• IOCP – I/O Control Processor

• FECP – Forwarding Engine Control Processor

• SPA – Shared Port Adapter

• SIP – SPA Interface Processor

• IOSd – IOS image that runs as a process on the RP

• FMAN – Forwarding manager (FMAN-RP, FMAN-FP)

• EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic

• IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP

19LTRARC-3500

Software Architecture


ESP

FECP

QFPCrypto

Assist.

interconn.

RP

CPU


SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

ASR1K Software ArchitectureRP

CPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

EO

BC

(1 G

bps)

ES

I (1

0-4

0 G

bp

s)

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ

DriversDriversDrivers

SIPIOCP

Linux Kernel

Chassis

Manager

SPA

SPA DriverSPA Driver

SPA Driver

SPA SPA

ES

I (1

0-4

0 G

bp

s)

I2C

21LTRARC-3500


RPCPU

IOS

Linux Kernel

ESP FECP

Linux Kernel

Crypto

Assist.

µµ

µBQS

µµ

µ

SIPIOCP

Linux Kernel

SPA


SPA Driver


SPA SPA

QFP

ES

I (1

0-4

0 G

bp

s)

Chassis Manager

Chassis Manager

Chassis

Manager

Forwarding Manager

ES

I (1

0-4

0 G

bp

s)

Forwarding Manager

Forwarding Manager (FMAN)

• FMAN on RP communicates with FMAN process on ESP• Distributed function

• Propagates control plane ops. to ESP• CEF tables, ACL’s, NAT, SA’s,…

• FMAN-FP communicates information back to FMAN-RP• e.g. statistics

• FMAN-RP pushes info back to IOS

• FMAN on active RP maintains state for both active & standby ESP’s• Facilitates NSF after re-start with bulk

download of state information

EO

BC

(1 G

bps)

I2C

FMAN-FP

ESP aka Forwarding Plane

FMAN-RP

22LTRARC-3500


PPE Microcode• Written in C

• proper features, no hack

• Runs on each thread of the PPE

• Processes packets

• run to completion

• assisted by various memories

• TCAM, DRAM,… various speeds

• Features applied via FIA

• Feature Invocation Array

• FIA per interface

• input FIA, output FIA

• drop FIA (Null interface)

RPCPU

IOS

Linux Kernel

ESP FECP

Linux Kernel

Crypto

Assist.

µµ

µBQS

µµ

µ

SIPIOCP

Linux Kernel

SPA


SPA Driver


SPA SPA

Chassis Manager

Forwarding Manager

Chassis Manager

Forwarding Manager

Chassis

Manager

ES

I (1

0-4

0 G

bp

s)

ES

I (1

0-4

0 G

bp

s)

EO

BC

(1 G

bps)

I2C

QFP

23

PPE Microcode runs here

QFP

Dispatcher

Packet Buffer


…

PPE1

PPE2

PPE3

PPE4

PPE5

PPE6

PPE7

PPE8

PPEN

BQS

LTRARC-3500

Resource Monitoring


The Vital Signs… RPCPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ


SIPIOCP

Linux Kernel

Chassis

Manager

SPA


SPA Driver

SPA SPA

Control Plane CPU’s

Data Plane CPU’s

Where does it hurt ?

25LTRARC-3500


Example: IOS Memory Usage vs IOSd RP Utilization

RPCPU

IOSChassis Manager

Forwarding Manager

Linux Kernel

asr-1k#show memory statistic

Load for five secs: 6%/1%; one minute: 5%; five minutes: 3%

Time source is NTP, 22:18:08.111 EDT Sat Apr 19 2014

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 300AE008 1713127140 564269356 1148857784 1066242316 992444168

lsmpi_io 963791D0 6295088 6294120 968 968 968

asr-1k#show process mem | inc BGP

523 0 2333028 51368 389076 313 313 BGP Router

asr-1k#show process cpu

…

asr-1k#show platform software process list RP active summary

…

Architecture : ppc

Memory (kB)

Physical : 4127744

Total : 3874516

Used : 2095636

Free : 1778880

asr-1k#show platform software process list RP active | inc fman

fman_rp 29015 27992 29015 S 20 136847360

Complex CLI, platform specific.

Additional information require connecting to the Linux shell

26LTRARC-3500


QFP Memory UtilizationIt is getting worse…

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.µ

µµ

BQSµ

µµ


TCAM DRAM DRAM

asr-1k#show platform hardware qfp active infrastructure exmem statistics

QFP exmem statistics

Type: Name: DRAM, QFP: 0

Total: 1073741824

InUse: 219466752

Free: 854275072

Lowest free water mark: 854005760

Type: Name: IRAM, QFP: 0

Total: 134217728

InUse: 8728576

Free: 125489152


Type: Name: SRAM, QFP: 0

Total: 32768

InUse: 15088

Free: 17680


asr-1k#show platform hardware qfp active tcam resource-manager usage

Load for five secs: 0%/0%; one minute: 1%; five minutes: 1%

Time source is NTP, 09:43:55.075 EDT Fri Apr 25 2014

QFP TCAM Usage Information

<snip>

Total TCAM Cell Usage Information

----------------------------------

Name : TCAM #0 on CPP #0

Total number of regions : 3

Total tcam used cell entries : 28

Total tcam free cell entries : 524260

Threshold status : below critical limit

asr-1k#show platform hardware qfp active infrastructure exmem statistics user

…

10 279092 284672 CEF

40 36441494 36458496 NAT

27LTRARC-3500


Resources - A Simplified View

28LTRARC-3500

asr-1k# show platform resources

Resource Usage Max Warning Critical State

RP0(ok, active) H

Control Processor 5.80% 100% 90% 95% H

DRAM 1814MB 3783MB 90% 95% H

ESP0(ok, active) H


DRAM 683MB 1962MB 90% 95% H

QFP H

DRAM 76244KB 524288KB 80% 90% H

IRAM 8817KB 131072KB 80% 90% H

SRAM 14KB 32KB 80% 90% H

TCAM 28cells 131072cells 80% 90% H

CPU Utilization 7.00% 100% 90% 95% H

ESP1(ok, standby) H


DRAM 683MB 1962MB 90% 95% H

QFP H

DRAM 76244KB 524288KB 80% 90% H

IRAM 8817KB 131072KB 80% 90% H

SRAM 14KB 32KB 80% 90% H

TCAM 28cells 131072cells 80% 90% H

CPU Utilization 0.00% 100% 90% 95% H

SIP0 H


DRAM 307MB 460MB 90% 95% H

SIP1 H


DRAM 160MB 460MB 90% 95% H

**State Acronym: H - Healthy, W - Warning, C – Critical

RPCPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ


SIPIOCP

Linux Kernel

Chassis

Manager

SPA


SPA Driver

SPA SPA

Introduced in IOS-XE 3.14


Other Show Commands ImprovementsImproves interaction with TAC

RPCPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

29LTRARC-3500

show processes memory platform

show processes cpu platform

show processes memory

show processes cpu

show memory platform

show memory


© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

Lab Access

LTRARC-3500

1. Use AnyConnect and log in to the dCloud environment.

2. Open the Cisco CLI Analyzer Telnet/SSH Client and log in

Master Password: cisco!123

3. Create a new session for each of the devices in your POD

• Click on “Devices”

• Enter the search term “LTRARC-3500” and press Enter

• Click on the device name to connect, use the below credentials:

Username: cisco

Password: cisco

• Click on “Devices” and connect to the remaining devices

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

IOS-XE Troubleshooting Lab Topology

Gig0/1

CSR2

10

.1.x

.x/1

6

.3.1

.3.3

ISP-2

Gig 0/1

Router5

Gig 0/1 .1.5

Router4

Router2

Router1

ISP-1

ISP-3

Router3

10.1.1.100

Client

10.3.3.100

Server

Gig 4

Gig 4

.1.1

10

.3.x

.x/1

6

10.10.10.10/32

.2

172.16.2.x /30

20.20.20.20/32

10.58.100.1/32

LTRARC-3500

Day in Life of Normal Packet


Ingress Packet Through SIP

…

ESPs

C2W

EV-FC

EV-RP

In ref clocks

Network clocks

SPA Agg.

SPA Aggregation

ASIC (Marmot)

Ingress

Scheduler

Egress

Buffer

Status

Ingress

Classifier

Egress buffers(per port)

Network

clock

distribution

IOCP

(SC854x SOC)

…

Ingress buffers(per port)

…

Interconnect

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl

Reset / Pwr Ctrl

SIP

SPA SPA

IOCPSPA

Aggreg.

intercon.

SPA

33LTRARC-3500


Crypto

FECP

Ingress Packet Through ESP

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

Interconnect

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2PPE2

34LTRARC-3500


Crypto

FECP

Packet Dispatched to PPE Core

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

35LTRARC-3500


Crypto

FECP

Packet Dispatched to PPE Core

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

Interconnect

PPE2

Th

read 1

Th

read 2

Th

read 3

Th

read 4

35LTRARC-3500


Crypto

FECP

Packet Dispatched to PPE Thread

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 1

Th

read 2

Th

read 4

Th

read 3

37LTRARC-3500


Crypto

FECP

FIA’s Applied on Packet by PPE Thread

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

38LTRARC-3500


Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3

X-Connect MPLS IPv4 IPv6

38LTRARC-3500


Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Input FIA

38LTRARC-3500


Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

TCP MSS adjust

VFR

Input FIA

38LTRARC-3500


Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

TCP MSS adjust

VFR

IP Unicast

IP Multicast

Packet For Us

Input FIA

38LTRARC-3500


Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

TCP MSS adjust

VFR

IP Unicast

IP Multicast

Packet For Us

Netflow

NAT

NBAR Classify

…

MQC Policing

MAC Accounting

Output ACL

Input FIA Output FIA

38LTRARC-3500


Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

PPE2

Thread 3


Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

TCP MSS Adjust

VFR

IP Unicast

IP Multicast

Packet For Us

Netflow

NAT

NBAR Classify

…

MQC Policing

MAC Accounting

Output ACL


44LTRARC-3500


Crypto

FECP

Leaving the PPE Thread

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE7

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

PPE2

Thread 3


Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

Dialer IDLE Rst

URD

IP Unicast

IP Multicast

Packet For Us

Netflow

NAT

NBAR Classify

…

MQC Policing

MAC Accounting

Output ACL


45LTRARC-3500

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS


Crypto

FECP

Packet Proceeding to BQS then SIP

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

Interconnect

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2PPE2

46LTRARC-3500


Egress Packet Through SIPESPs

C2W

EV-FC

EV-RP

In ref clocks

Network clocks

SPA Agg.

SPA Aggregation

ASIC (Marmot)

Ingress

Scheduler

Egress

Buffer

Status

Ingress

Classifier

Egress buffers(per port)

Network

clock

distribution

IOCP

(SC854x SOC)

…

Ingress buffers(per port)

…

Interconnect

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl

Reset / Pwr Ctrl

SIP

SPA SPA

IOCPSPA

Aggreg.

intercon.

SPA

47LTRARC-3500


ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

Punt Path: From QFP to Internal Destination

RP

CPU


Midplane

48LTRARC-3500

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.


ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

Midplane

PPE2

Thread 3

Punt Path: From QFP to Internal Destination

RP

CPU


49LTRARC-3500

• Punt to RP for us control

• Punt to RP for us data

• Punt to RP cause “X”…

• Punt to Recycle

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn. internal0/0/rp:0

RP has its own dedicated internal interface on QFP: internal0/0/rp:0

Recycle path interface name on QFP: internal0/0/recycle:0


Midplane

ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

Inject Path: From RP via QFP to the network

RP

CPU


50LTRARC-3500

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.


ESP

FECP

QFPCrypto

Assist.

interconn.

PPE BQS

Midplane

PPE2

Thread 3

Inject Path: Recycling packet via QFP to the network

RP

CPU


51LTRARC-3500

• Recycle path

SIP

SPA SPA

IOCPSPA

Aggreg.

interconn.

Packet-tracer and FIA Debugger


Crypto

FECP

The Packet Tracer and FIA Debugger

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

53LTRARC-3500



Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Input FIA

53LTRARC-3500



Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Input FIA

Pak Match ?

Condition determines

packets to be traced

53LTRARC-3500



Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Input FIA

Pak Match ?

Packet # 16Condition determines


53LTRARC-3500



Crypto

FECP


RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3


Input ACL

MQC Classify

NAT

PBR

IP Unicast

Output ACL

NAT

Encaps

Crypto


Pak Match ?

Packet # 16

Input ACL

MQC Classify

NAT

PBR

Output ACL

NAT

Encaps

Crypto

Optionally, FIA actions can logged per packet

System can capture several packets flows

Packet flows can be reviewed in show commands



Statistics and final action will be

collected (matched packets dropped,

punted to RP, forwarded to output

interface …)

Optionally match

on the egress FIA

53LTRARC-3500



Packet-Trace: AccountingAccounting keeps a count of all pactrac interesting packets that enter and leave the “packet processor”. There are three basic count groups.

Summary counts

• Packets Matched –packets that matched conditions

• Packets Traced – packets that were traced

Arrival counts

• Ingress – packets entering via external interfaces

• Inject* – number of packets seen as injected from control plane


Packet-Trace: AccountingDeparture counts

• Forward – number of packets scheduled/queued for delivery

• Punt* – number of packets punted to control plane

• Drop* – number of packets specifically dropped by packet processing

• Consume – number of packets consumed during packet process (e.g. ping request)

* Per reason/code counts are maintained for Inject, Punt and Drop.


Packet-Trace: Summary DataWhen enabled, summary data is collected for a specified number of packets and includes:

• Packet number (pactrac specific packet number)

• Input interface

• Output interface

Final packet state and any punt/drop/inject codes

Collecting summary data uses little performance over the normal packet processing. An example usage may be to isolate which interfaces are dropping traffic so more detailed inspection can be used after applying interface specific conditions.


Packet-Trace: Path DataPath data may be collected per packet for a limited number of packets and is made up of different types of data as follows:

• Common path data (e.g. IP tuple)

• Feature specific data (e.g. NAT)

• Feature Invocation Array (FIA) trace – optionally enabled

• Copy of all or part of the incoming and/or outgoing packet – optionally enabled

Capturing path data has the greatest impact on packet processing* capability specifically FIA trace and packet copy.

• FIA tracing creates many path data entries costing instructions and DRAM writes

• Packet copy creates many DRAM read/writes

*Recall the packet-trace will only affect the performance of packets traced (i.e. those matched by the user provided conditions)


Conditionally Matching PacketsIdentifying Interesting Packets

asr-1k# debug platform condition ?

debug platform condition ?

both Simultaneous ingress and egress debug

egress Egress only debug

…

ingress Ingress only debug

interface Set interface for conditional debug

ipv4 Debug IPv4 conditions


mpls Debug MPLS conditions

…

asr-1k#debug platform condition ingress

asr-1k#debug platform condition interface gig0/0/3 ingress

asr-1k#debug platform condition ipv4 10.0.0.1/32 both

asr-1k#debug platform condition ipv4 access-list 100 egress

asr-1k#debug platform condition mpls 10 1 ingress

Match all ingress packets

Match MPLS packets with

top ingress label 10

Match all ingress packets on interface gig0/0/3

Match in & out packets with source or

destination 10.0.0.1

Match egress packets passing

access-list 100

62LTRARC-3500


Activating the Packet TracerFollowing packets through IOS-XE – Basic Statistics

asr-1k# debug platform condition interface gig0/0/0 ingress

asr-1k# debug platform condition start

asr-1k# debug platform packet-trace enable

asr-1k# … !send traffic

asr-1k# show platform packet-trace statisticsPackets Summary

Matched 102

Traced 0

Packets Received

Ingress 12

Inject 90

Count Code Cause

90 9 QFP ICMP generated packet

Packets Processed

Forward 12

Punt 0

Drop 90

Count Code Cause

13 92 Ipv4Null0

17 47 FirewallInvalidZone

60 184 FirewallL4

Consume 0

102 packets were matched by

the condition

12 packets were forwarded

90 packets were dropped

13 packets were dropped

due to no route

17 packets were dropped due to

absence of zone pair60 packets dropped by L4 inspection

(e.g. receiving window)

asr-1k# debug platform packet-trace ?

copy Copy packet data

drop Trace drops only

enable Enable packet trace

packet Packet count

The packet tracer follows a

set of packets in details

through the FIA

Extraneous command -

was suppressed in 16.3

63LTRARC-3500


Packet Tracer – Tracing Packets…The fate of 16 packets



asr-1k# debug platform packet-trace packet 16



asr-1k# show platform packet-trace summary

Pkt Input Output State Reason

0 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)




4 INJ.7 Gi0/0/2 FWD

5 INJ.7 Gi0/0/2 FWD


7 INJ.7 Gi0/0/2 FWD

8 …

Automatically stops tracing

after 16 packets

16 packets were traced; we

can zoom in

INJ.7: Packet injected by the RP

internal0/0/rp:0: Packet punted to the RP

Extraneous command -

was suppressed in 16.3

64LTRARC-3500


Packet Tracer – Tracing Packets…The fate of an individual packet

asr-1k# show platform packet-trace packet 1

Packet: 1 CBUG ID: 109056985

Summary

Input : GigabitEthernet0/0/2

Output : internal0/0/rp:0

State : PUNT 55 (For-us control)

Timestamp

Start : 334771580191282 ns (04/29/2014 08:01:38.017738 UTC)

Stop : 334771580487612 ns (04/29/2014 08:01:38.018035 UTC)

Path Trace

Feature: IPV4

Source : 17.0.0.196

Destination : 172.18.0.1

Protocol : 50 (ESP)

Feature: IPSec

Action : DECRYPT

SA Handle : 753

SPI : 0x30ba5940

Peer Addr : 17.0.0.196

Local Addr: 172.18.0.1

Zooming on packet 1

Only major features

are shown

Feature specific details are

displayed

65LTRARC-3500


Packet Tracer – Tracing Packets... even keeping a copy of the packet if necessary




asr-1k# debug platform packet-trace copy packet both [l2 | l3 | l4]



asr-1k# show platform packet-trace packet 1


Summary




Path Trace

Feature: IPV4

Feature: IPSec

Packet Copy In

45c00088 c5ee0000 ff32346f 11000313 ac120001 d4b46317 0000017c 68a60265

0ef58135 650e2341 15cf6e81 dd434455 b42efef8 c6cf5ab1 44ad3f98 b165c3d5

Packet Copy Out

45c0003c 00000000 015804f4 c0ab1301 e000000a 0205efc8 00000000 00000000

00000000 0000000a 0001000c 01000100 0000000f 00040008 0a000200

Keep a copy of the packet in

ingress and egress of the ESP

(before and after the FIA)

Display the stored packet copy

Can store L2, L3 or L4…

pick-a-choose

66LTRARC-3500


Packet Tracer – Tracing Packets…The fate of a single packet… even more more more details

asr-1k# show platform packet-trace packet 1 decode


Summary




Path Trace

Feature: IPV4

Feature: IPSec

Packet Copy In

45c00088 c5ee0000 ff32346f 11000313 ac120001 d4b46317 0000017c 68a60265

0ef58135 650e2341 15cf6e81 dd434455 b42efef8 c6cf5ab1 44ad3f98 b165c3d5

IPv4

Version : 4

Header Length : 5

ToS : 0xc0

Total Length : 136

Identifier : 0xc5ee

IP Flags : 0x0

Frag Offset : 0

TTL : 255

Protocol : 50 (ESP)

Header Checksum : 0x346f

Source Address : 17.0.3.19

Destination Address : 172.18.0.1

ESP

SPI : 0xd4b46317

Sequence Number : 0x0000017c

...

Decode the stored packet copy

Here showing the input copy

(output copy follows)

67LTRARC-3500


Packet Tracer – Focus on DropsDropped packets – nothing else




asr-1k# debug platform packet-trace drop [code <dropcode>]



asr-1k# debug platform condition stop

asr-1k# show platform packet-trace summary

Pkt Input Output State Reason

0 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)








8 …

Only save dropped packets

Focus on specific drop codes

(find codes in packet-trace statistics)

Stop tracing before dumping the

summary (code limitation)

Admire dropped packets… real close

asr-1k#show platform packet-trace packet 1


Summary


Output : GigabitEthernet0/0/2

State : DROP 53 (IpsecInput)

Timestamp

Start : 361426338620013 ns (04/29/2014 15:25:52.785406 UTC)

Stop : 361426338684993 ns (04/29/2014 15:25:52.785471 UTC)

Path Trace

Feature: IPV4

Source : 17.0.1.34

Destination : 172.18.0.1

Protocol : 50 (ESP)

Packet Copy Out

002304bb 72020007 7dfbe301 080045c0 0088d135 0000fe32 2c191100 0122ac12

0001085e 1d620000 00c8172c e8010c3e 44726e6f 3eb231d5 166298c1 f519313c

For drops, condition is optional…

68LTRARC-3500


Packet Tracing – Basic and FIA-TRACE

asr1000# show platform hardware qfp active interface if-name

gig1

General interface information

Interface Name: GigabitEthernet1

Interface state: VALID

Platform interface handle: 7

QFP interface handle: 6

…

Protocol 0 - ipv4_input

FIA handle - CP:0x2fccfe0 DP:0xe73998c0

[…]

IPV4_INPUT_DST_LOOKUP_ISSUE (M)

IPV4_INPUT_ARL_SANITY (M)

CBUG_INPUT_FIA

DEBUG_COND_INPUT_PKT

asr1000#show platform packet-trace packet 0


Summary

Input : GigabitEthernet1

Output : GigabitEthernet3

State : FWD

Timestamp

Start : 5456699323393 ns (07/11/2016 23:30:28.244810 UTC)

Stop : 5456699556099 ns (07/11/2016 23:30:28.245043 UTC)

Path Trace

Feature: IPV4


Output : <unknown>

Source : 192.168.3.1

Destination : 192.168.255.167

Protocol : 50 (ESP)

Feature: FIA_TRACE


Output : <unknown>

Entry : 0x8139f260 - DEBUG_COND_INPUT_PKT

Lapsed time : 9680 ns

asr1000#show platform packet-trace packet 1


Summary


Output : GigabitEthernet3

State : FWD

Timestamp

Start : 5331698002827 ns (07/11/2016 23:28:23.187027 UTC)

Stop : 5331698159842 ns (07/11/2016 23:28:23.187184

UTC)

Path Trace

Feature: IPV4


Output : <unknown>

Source : 192.168.3.1

Destination : 192.168.255.167

Protocol : 50 (ESP)

Features Pack Tracer Pack Tracer w/ FIA-TRACE

69LTRARC-3500


Packet Tracing – Basic and FIA-TRACE (II)

IPV4_INPUT_DST_LOOKUP_CONSUME (M)

IPV4_INPUT_ACL

IPV4_INPUT_FOR_US_MARTIAN (M)

Feature: FIA_TRACE


Output : <unknown>

Entry : 0x813a5554 -

IPV4_INPUT_DST_LOOKUP_CONSUME


Feature: FIA_TRACE


Output : <unknown>

Entry : 0x80f67140 - IPV4_INPUT_ACL


Feature: FIA_TRACE


Output : <unknown>

Entry : 0x813a5558 - IPV4_INPUT_FOR_US_MARTIAN



70LTRARC-3500


Packet Tracing – Basic and FIA-TRACE (III)

IPV4_INPUT_STILE_LEGACY Feature: CFT

API : cft_handle_pkt

packet capabilities : 0x0000008c

input vrf_idx : 0

calling feature : STILE

direction : Input

triplet.vrf_idx : 0

triplet.network_start : 0x00000000

triplet.triplet_flags : 0x00000000

triplet.counter : 0

cft_bucket_number : 2120447

cft_l3_payload_size : 100

cft_pkt_ind_flags : 0x00000000

cft_pkt_ind_valid : 0x00000935

tuple.src_ip : 192.168.3.1

tuple.dst_ip : 192.168.255.167

[…]

Feature: NBAR

Packet number in flow: N/A

Classification state: Final

Classification name: ipsec

Classification ID: [CANA-L7:9]

Number of matched sub-classifications: 0

Number of extracted fields: 0

Is PA (split) packet: False

TPH-MQC bitmask value: 0x0

Feature: FIA_TRACE


Output : <unknown>

Entry : 0x80fa0f88 - IPV4_INPUT_STILE_LEGACY


Feature: CFT

API : cft_handle_pkt

packet capabilities : 0x0000008c

input vrf_idx : 0

calling feature : STILE

direction : Input

triplet.vrf_idx : 0

triplet.network_start : 0x00000000

triplet.triplet_flags : 0x00000000

triplet.counter : 0

cft_bucket_number : 2120447

cft_l3_payload_size : 100

cft_pkt_ind_flags : 0x00000000

cft_pkt_ind_valid : 0x00000935

tuple.src_ip : 192.168.3.1

tuple.dst_ip : 192.168.255.167

[…]

Feature: NBAR

Packet number in flow: N/A

Classification state: Final

Classification name: ipsec

[…]


71LTRARC-3500


Packet Tracing – Basic and FIA-TRACE (IV)

IPV4_INPUT_QOS

IPV4_INPUT_VFR

IPV4_NAT_INPUT_FIA

IPV4_INPUT_LOOKUP_PROCESS (M)

Feature: QOS

Direction : Ingress

Action : SET

Fields : DSCP

Feature: FIA_TRACE


Output : <unknown>

Entry : 0x813a6fe4 - IPV4_INPUT_QOS


Feature: FIA_TRACE


Output : <unknown>

Entry : 0x813a5574 - IPV4_INPUT_VFR


Feature: FIA_TRACE


Output : <unknown>

Entry : 0x80f99600 - IPV4_NAT_INPUT_FIA


Feature: FIA_TRACE


Output : <unknown>

Entry : 0x813ae9b0 - IPV4_INPUT_LOOKUP_PROCESS


Feature: QOS

Direction : Ingress

Action : SET

Fields : DSCP


72LTRARC-3500


Packet Tracing – Basic and FIA-TRACE (V)

IPV4_INPUT_IPOPTIONS_PROCESS (M)

IPV4_INPUT_GOTO_OUTPUT_FEATURE (M)

Feature: FIA_TRACE


Output : <unknown>

Entry : 0x813a557c - IPV4_INPUT_IPOPTIONS_PROCESS


Feature: FIA_TRACE

Input : Virtual-Access19

Output : <unknown>

Entry : 0x813ae9b4 -

IPV4_INPUT_GOTO_OUTPUT_FEATURE


73LTRARC-3500

Debugging Strategies


Everyday Situations

Traffic did not reach its target !

What happened to that packet ?

Why did that happen ?

75LTRARC-3500


Everyday Situations

Which feature went wrong ?

What went wrong in the feature ?

NATZBFIPsec

Routing

WAASOTV

SNMP

Ordering

Ambiguity

Config

Traffic

issue

Bug

MemoryPerformance

First

76LTRARC-3500


Everyday Situations

What went wrong in the feature ?

Ordering

Ambiguity

Config

Traffic

issue

Bug

MemoryPerformanc

e

Second

Config Ordering

AmbiguityTraffic

issue

MemoryPerformance

77LTRARC-3500


Using statistics for troubleshooting packet drops

• SPA

• show interfaces <interface-name>

• show interfaces <interface-name> accounting

• show interfaces <interface-name> stats

• SIP

• show platform hardware port <slot/card/port> plim statistics

• show platform hardware subslot {slot/card} plim statistics

• show platform hardware slot {slot} plim statistics

• show platform hardware slot {0|1|2} plim status internal

• show platform hardware slot {0|1|2} serdes statistics

• RP

• show platform hardware slot {r0|r1} serdes statistics

• show platform software infrastructure lsmpi

• ESP

• show platform hardware slot {f0|f1} serdes statistics

• show platform hardware slot {f0|f1} serdes statistics internal

• show platform hardware qfp active bqs 0 ipm mapping

• show platform hardware qfp active bqs 0 ipm statistics channel all

• show platform hardware qfp active bqs 0 opm mapping

• show platform hardware qfp active bqs 0 opm statistics channel all

• show platform hardware qfp active statistics drop [detail]

• show platform hardware qfp active interface if-name <Interface-name> statistics

• show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_

• show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude _0_

• show platform hardware qfp active infrastructure punt statistics type inject-drop | exclude _0_

• show platform hardware qfp active infrastructure punt statistics type global-drop | exclude _0_

• show platform hardware qfp active infrastructure bqs queue output default all

• show platform hardware qfp active infrastructure bqs queue output recycle all

Not easy… not very practical either.

Let’s dig deeper before making it simpler

78LTRARC-3500


Debugging Strategies to Date

IOS Control Plane

• ACL + show access-list,…

• show interface / ip route / bgp …

Platform Control Plane

• ESP “stuff”

• e.g. show platform … hard to remember

Data Plane

• ESP “stuff”

• More arcane show platform …

Top D

ow

n

Let’s

change

that!!

Rock bottom

79LTRARC-3500

Troubleshooting Tools and Capabilities

Understanding and Extracting ESP Logs


ESP Tracing aka LoggingRP

CPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

EO

BC

(1 G

bps)

ES

I (1

0-4

0 G

bp

s)

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ


SIPIOCP

Linux Kernel

Chassis

Manager

SPA


SPA Driver

SPA SPA

ES

I (1

0-4

0 G

bp

s)

I2C

Mounted NFS

ESP logs are committed

here at regular intervals

TEMP RAM FS

ESP logs are first written

here (efficiency)

NFS Shared Disk

Hard disk is really here

TEMP RAM FS

RP logs are first written

here (efficiency)

82LTRARC-3500


Important LogsRP

CPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

EO

BC

(1 G

bps)

ES

I (1

0-4

0 G

bp

s)

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ


SIPIOCP

Linux Kernel

Chassis

Manager

SPA


SPA Driver

SPA SPA

ES

I (1

0-4

0 G

bp

s)

I2C

Under /harddisk/tracelogs/

fman-fp_R0.log.<timestamp>

cpp_cp_F[0|1]-0.log.<timestamp>

Under /harddisk/tracelogs

fman_fp_F[0|1]-0.log

cpp_cp_F[0|1]-0.log

fman_rp_R[0|1]-0.log

fman_rp_R[0|1]-0.log.<timestamp>



83LTRARC-3500


What log files are important?

• Important log files to get for security issues:• fman_rp_R[0|1].log (under /tmp/rp/trace directory on RP)

• fman-fp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP

• cpp_cp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP)

• All these logs get rotated and are copied to /harddisk/tracelogs directory on active RP.

• Look for the relevant log files depending on the time of the failure

• By default, all ERR messages are logged should be the first things to look for

84LTRARC-3500


Example log files

My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*Directory of harddisk:/tracelogs/cpp_cp_F0*Directory of harddisk:/tracelogs/3768365 -rwx 1048934 Jan 6 2014 18:20:16 +00:00 cpp_cp_F0-0.log.7133.201401061820153768330 -rwx 551643 Jan 7 2014 09:27:51 +00:00 cpp_cp_F0-0.log.7133.201401070927513768335 -rwx 1048901 Jan 7 2014 08:56:44 +00:00 cpp_cp_F0-0.log.7133.2014010708564339313059840 bytes total (30680653824 bytes free)

The timestamp…

85LTRARC-3500


Rotating the log files

My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*

Directory of harddisk:/tracelogs/cpp_cp_F0*

Directory of harddisk:/traceMy-ASR1000-2#test platform software trace slot rp active forwarding-manager rotate

Rotated file from: /tmp/rp/trace/stage/fman_rp_R0-0.log.13836.20140107094754, Bytes: 0, Messages: 6535

My-ASR1000-2#test platform software trace slot FP active cpp-control-process rotate

Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7133.20140107093650, Bytes: 154027, Messages: 786

My-ASR1000-2#test platform software trace slot FP active forwarding-manager rotate

Rotated file from: /tmp/fp/trace/stage/fman-fp_F0-0.log.8247.20140107093738, Bytes: 20170, Messages: 210

OR use

My-ASR1000-2#request platform software trace rotate all

Does not show the rotated file names w/

time stamp have to hunt them down

86LTRARC-3500


Platform logs… ComplexityRP

CPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

EO

BC

(1 G

bps)

ES

I (1

0-4

0 G

bp

s)

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ


SIPIOCP

Linux Kernel

Chassis

Manager

SPA


SPA Driver

SPA SPA

ES

I (1

0-4

0 G

bp

s)

I2C

Under /harddisk/tracelogs/



Under /harddisk/tracelogs


cpp_cp_F[0|1]-0.log





87LTRARC-3500


New logging framework: Show logging processRP

CPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

EO

BC

(1 G

bps)

ES

I (1

0-4

0 G

bp

s)

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ


SIPIOCP

Linux Kernel

Chassis

Manager

SPA


SPA Driver

SPA SPA

ES

I (1

0-4

0 G

bp

s)

I2C


cpp_cp_F[0|1]-0.log


88LTRARC-3500

Show logging process <process name> internal

#csr1000v-1# show logging process fman internal

excuting cmd on chassis local ...

Collecting files on current[local] chassis.

Total # of files collected = 4

Decoding files:

/harddisk/tracelogs/tmp_trace/fman_fp_F0-0.21047_0.20180109071524.bin: DECODE(592:0:592:10)

/harddisk/tracelogs/tmp_trace/fman_rp_R0-0.14852_0.20180109071523.bin: DECODE(21:0:21:11)

/harddisk/tracelogs/tmp_trace/fman_rp_pmanlog_R0-0.14682_0.20180109071455.bin: DECODE(25:0:25:1)

/harddisk/tracelogs/tmp_trace/fman_fp_image_pmanlog_F0-0.20738_0.20180109071508.bin: DECODE(28:0:28:1)

<……decoded files>



New logging framework: Show logging profileRP

CPU

IOS

Chassis Manager

Forwarding Manager

Linux Kernel

EO

BC

(1 G

bps)

ES

I (1

0-4

0 G

bp

s)

ESP FECP

Linux Kernel

Chassis Manager

Forwarding Manager

QFPCrypto

Assist.

µµ

µBQS

µµ

µ


SIPIOCP

Linux Kernel

Chassis

Manager

SPA


SPA Driver

SPA SPA

ES

I (1

0-4

0 G

bp

s)

I2C


cpp_cp_F[0|1]-0.log





89LTRARC-3500

Show logging profile <profile name> internal

csr1000v-1# show logging profile iwan internal

executing cmd on chassis local ...

Collecting files on current[local] chassis.

Total # of files collected = 16

Decoding files:2018/01/09 07:14:55.770 {fman_rp_pmanlog_R0-0}{1}: [fman_rp_pmanlog] [14682]: (note): gdb port 9905 allocated

2018/01/09 07:14:55.812 {fman_rp_pmanlog_R0-0}{1}: [fman_rp_pmanlog] [14682]: (note): swift_repl port 8005 allocated

2018/01/09 07:14:55.882 {fman_rp_pmanlog_R0-0}{1}: [fman_rp_pmanlog] [14682]: (info): (std):

/tmp/sw/rp/0/0/rp_security/mount/usr/binos/conf/pman.sh: line 424: sigusr1_func: readonly function

2018/01/09 07:14:55.902 {fman_rp_pmanlog_R0-0}{1}: [fman_rp_pmanlog] [14682]: (note): process scoreboard

/tmp/rp/process/fman_rp%rp_0_0%0 fman_rp%rp_0_0%0.pid is 1458

22018/01/09 07:14:55.902 {fman_rp_pmanlog_R0-0}{1}: [fman_rp_pmanlog] [14682]: (note): fman_rp%rp_0_0%0.gdbport is 9905

2018/01/09 07:14:55.902 {fman_rp_pmanlog_R0-0}{1}: [fman_rp_pmanlog] [14682]: (note): fman_rp%rp_0_0%0.swift_replport is 8005


Embedded Packet Capture


The Embedded Packet CaptureOne way of capturing packets…

Device# show monitor capture mycap buffer dump

0

0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E.

0010: 00300000 00000111 CFDC091D 0002E000 .0..............

0020: 000207C1 07C1001C 802A0000 10030AFA .........*......

0030: 1D006369 73636F00 0000091D 0001 ..example.......

1

0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F.

0010: 00200000 00000102 44170000 0000E000 . ......D.......

0020: 00019404 00001700 E8FF0000 0000 ..............

2

0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E.

0010: 00300000 00000111 CFDB091D 0003E000 .0..............

0020: 000207C1 07C1001C 88B50000 08030A6E ...............n

0030: 1D006369 73636F00 0000091D 0001 ..example.......

Device# monitor capture mycap start

Device# monitor capture mycap access-list v4acl

Device# monitor capture mycap limit duration 1000

Device# monitor capture mycap interface GigabitEthernet 0/0/1 both

Device# monitor capture mycap buffer circular size 10

Device# monitor capture mycap start

Device# monitor capture mycap export tftp://10.1.88.9/mycap.pcap

Device# monitor capture mycap stop Shows whether packets have been received or sent

Shows what packets look like

Requires hex dump analysis or export to decoder (sniffer)

Does not tell us what happened to the packet

Excellent tool but insufficient in many cases

http://www.cisco.com/en/US/docs/ios-

xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-capture-

xe.html

91LTRARC-3500


© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 92LTRARC-3500

Embedded Packet Capture

• EPC added to FIA

• Beginning of ingress FIA

• End of egress FIA

• Matched packets are copied

• Copied packets get punted to RP

• Original packets processed as usual

• Capture buffer on RP can be exported to .pcap file

TFTP Server

Router

Capture Buffer

Gi0/0/1

Export

Capture point

Gi0/0/2

Conditional Feature Debugging


Crypto

FECP

FIA’s Applied on Packet by PPE thread

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

BRKCRS-3147 94

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS


Crypto

FECP

FIA’s Applied on Packet by PPE thread

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

PPE2

Thread 3

X-Connect L2 Switch IPv4 IPv6 MPLS

Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

TCP MSS Adjust

VFR

IP Unicast


...

OUTPUT_INSPECTOUTPUT_INSPECT

BRKCRS-3147 94

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS


Crypto

FECP

Inside Output Threat Inspect

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

96

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

PPE2

Thread 3

IPv4 IPv6 MPLSL2 SwitchX-Connect

OUTPUT_INSPECT

Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

Dialer IDLE Rst

URD

IP Unicast

Output FIAInput FIA

...

OUTPUT_INSPECT

IPV4 OUTPUT

INSPECT

Session DB

Policy Selection Session Lookup(precise + imprecise) Classify Traffic

Create Session

L4 Inspection

L7 Parse

L7 InspectionImprecise Channel

Creation

Input

Output

Pass

Drop

Miss

Hit

Inspect

Pass

Drop

TCAM

BRKCRS-3147


Crypto

FECP

Inside Output Threat Inspect

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

ESP

FECP

QFPCrypto

Assist.

intercon.

PPE BQS

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

PPE2

Thread 3


Netflow

Input ACL

NBAR Classify

MQC Classify

…

NAT

PBR

Dialer IDLE Rst

URD

IP Unicas

t


...

OUTPUT_INSPECTOUTPUT_INSPECT

IPV4 OUTPUT

INSPECT

Session DB

Policy Selection Session Lookup(precise + imprecise) Classify Traffic

Create Session

L4 Inspection

L7 Parse

L7 InspectionImprecise Channel

Creation

Input

Output

Pass

Drop

Miss

Hit

Inspect

Pass

Drop

µIDB input+output Zone Pair Policy

Using Session DB in DRAM

Imprecise lookup only for initial

packets (syn…)

If Action = Inspect, create

session flow in DB

PDU reassembly, parsing

(HTTP GET, POST,…)

Action MappingChild session creation (data flow

from FTP, RTP flow from SIP,…)

TCAM

Match each class-map in

policy (ACL’s in TCAM)

BRKCRS-3147 97


Crypto

FECP

The Packet Tracer and Conditional Debugger

RPs RPs RPsESP SIPs

QFP Complex


DRAM

Part Len / BW

SRAM

SA table

DRAM

Dispatcher

Packet Buffer

DDRAM

Boot Flash

(OBFL,…)

JTAG Ctrl

EEPROM

Temp Sensor

Reset / Pwr Ctrl


…

PPE1 PPE3 PPE4 PPE5

PPE6 PPE7 PPE8 PPEN

BQS

Reset / Pwr Ctrl

SPI Mux

PPE2

Interconnect

PPE2

Th

read 2

Th

read 1

Th

read 4

Th

read 3

PPE2

Thread 3


Input ACL

MQC Classify

NAT

PBR

IP Unicast

Output ACL

NAT

Encaps

Crypto


Ingress Match ?

Packet # 16

Input ACL

MQC Classify

NAT

PBR

Output ACL

NAT

Encaps

Crypto



The packet tracer collects statistics

and final action (matched packets

dropped, punted to RP, forwarded to

output interface …)

Output ACL

NAT

Encaps

Crypto

If feature conditional debugger is

activated, these blocks will be

debugged

Our focus now

98LTRARC-3500



Conditionally Matching PacketsStep 1 – Identifying packets


debug platform condition ?

both Simultaneous ingress and egress debug

egress Egress only debug

…

ingress Ingress only debug

interface Set interface for conditional debug



mpls Debug MPLS conditions

…

asr-1k#debug platform condition ingress

asr-1k#debug platform condition interface gig0/0/3 ingress

asr-1k#debug platform condition ipv4 10.0.0.1/32 ingress

asr-1k#debug platform condition ipv4 access-list 100 ingress

asr-1k#debug platform condition mpls 10 1 ingress

Match all ingress packets

Match MPLS packets with

top ingress label 10

Match all ingress packets on interface gig0/0/3

Match ingress packets with source or

destination 10.0.0.1

Match ingress packets passing

access-list 100

99LTRARC-3500


Feature DebuggingStep 2 – Define feature(s) to troubleshoot

asr-1k# debug platform condition feature ?

acl ACL feature

alg ALG feature

fw FW feature

ipsec IPSEC feature

nat NAT feature

nat64 NAT64 feature

…

acl ACL feature

alg ALG feature

appnav AppNav feature

atm ATM feature

atom ATOM feature

bridge-domain Layer2 bridging feature

cent CENT feature

cft CFT feature

cxsc CXSC feature

dpss DPSS feature

evc EVC feature

fw FW feature

ipsec IPSEC feature

lisp LISP feature

multicast multicast feature

nat NAT feature

nat64 NAT64 feature

nbar NBAR feature

overlay overlay feature

qos QOS feature

subscriber Subscriber feature

tcp TCP feature

vpls VPLS feature

Many features are supported but

focus is on NAT, ZBF and FW at

the moment

100LTRARC-3500


Feature Debugging (cont.)

Step 2 (cont.) – Define feature submodes to be troubleshot

asr-1k# debug platform condition feature fw dataplane submode ?

alg-inspect Debug firewall ALG inspect information

all Debug firewall all information

detail Debug firewall detail

drop Debug firewall drop information

event Debug firewall event information

ha Debug firewall HA information

layer4 Debug firewall Layer 4 information

level Debug level information

policy Debug firewall policy information

asr-1k# debug platform condition feature fw dataplane submode drop layer4 policy

Multiple submodes can be active at once

101LTRARC-3500


Start & Stop Conditional DebuggingStep 3 – Start marking the packets (internally) and debug features


…

start Start conditional debug

stop Stop conditional debug

asr-1k#debug platform condition start

asr-1k#...

asr-1k#debug platform condition stop

After this, analyze the debugs…

Debugs won’t show on

console (yet)

102LTRARC-3500

Wrapping up…


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#LTRARC-3500


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Tech Circle

• Meet the Engineer 1:1 meetings

• Related sessions

106LTRARC-3500

Thank you

Date post:	18-Mar-2019
Category:	Documents
Upload:	dangkhue
View:	260 times
Download:	2 times

IOS-XE Troubleshooting · ESP FECP QFP Crypto Assist. interconn. PPE BQS ESP FECP Crypto Assist....

Documents