Cisco Live! :: Cisco ASR 9000 Architecture :: BRKARC-2003 | Las Vegas 2017

Cisco ASR 9000 System Architecture

Yongzhong Peng

Technical Marketing Engineer, SP Routing Infrastructure

BRKARC-2003

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to chat with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKARC-2003Cisco Spark spaces will be available until July 3, 2017.

• ASR9000 Products Introduction

• ASR9000 Distributed Control Plane

• ASR9000 Data Packet Processing

• ASR9000 QoS Architecture

• IOS-XR & IOS-XR 64 Bit

• Conclusion

Agenda

ASR 9000 Products Introduction


Compact & Powerful

Access/AggregationFlexible Service Edge

High Density Service Edge

and Core

• Small footprint with full IOS-XR for distributed environments

• Optimized for ESE and MSE with high M-D scale for medium to large sites

• Scalable, ultra high density service routers for large, high-growth sites

One Platform, One OS, One Family

Fixed 2RU

240 Gbps

2 LC/6RU

16 Tbps

8 LC/21RU

14 Tbps

10 LC/30RU

80 Tbps

20 LC/44RU

160 Tbps

4 LC/10RU

7 Tbps

ASR 9904

ASR 9001

ASR 9006

ASR 9010

ASR 9912

ASR 9922

MSE E-MSE Peering P/PE Mobility Broadband

nV Satellites ASR 9000v, NCS5000

CE

8 LC/21RU

64 Tbps

ASR 9910

4 LC/14RU

32 Tbps

ASR 9906ASR 9901

Cisco ASR 9000 System Comprehensive Portfolio

BRKARC-2003 6


Form Factor, Ports & BW

• Combination of 1G, 1/10G and 100G ports

• 2 RU box with 2 Tomahawk NPU(Ironman is 2 RU with 2 Typhoons)

• Depth of ~23 inches,(Ironman is 18”)

• 456G Duplex BW(Ironman is 120G Duplex)

• Fixed ports available; no MPAs

• 42 ports on the faceplate : 16X1G, 24X1/10G, 2X100G(QSFP28)

• 1G ports : LAN & MACSEC10G ports: LAN & MACSEC100G port : LAN & MACSEC

Ports / Port Density

• Redundant Power & Fan-trays

• Front to back Airflow

• NEBS, EMC Compliant

• All ports/power cabling on front plate; fan trays on backside

• Typical Power Consumed : 1200W

Mechanicals & Commons

• 64 bit XR only

• PAYG mode for 120G,240G,360G and 456G

• Full feature parity with Tomahawk feature-set

SW & Licensing

ASR9901 Overview

BRKARC-2003 7

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8BRKARC-2003

ASR 9901 Port mapping


Cisco ASR 9904 Overview

Feature Description

Total Capacity 16T

Capacity per Slot 4T

Slots 4 slots - 2 Line Cards and 2 RSPs

Rack size 6RU

Power 1 Power Trays, 4 Power Modules

2.1 KW DC / 3.0 KW AC supplies

Fan Side to Side Airflow, Front-to-Back Optional

1 Fan Tray, FRU

RSPs Integrated Fabric, 1+1 Redundancy

Line cards Tomahawk

Typhoon

SIP700, VSM

Front-to-back air flow with air flow baffles, 10RU

BRKARC-2003 9


Feature Description

Capacity 32T

Capacity /Slot 4T


Size 14RU

Architecture Mid-plane

Power

1 Power Tray

4.4 KW DC supplies

6.0 KW AC supplies

FanFront-to-Back Airflow

2 Fan Trays, FRU


Fabric Cards

5 Fabric cards in the back for additional capacity

230G per FC at FCS

6+1 Redundancy – 2 RSP’s integrated fabric + 5 FCs

Line cardsTomahawk

Not supported LCs – SIP700, VSM and Typhoon

IOS-XR 64 Bit

Front View Rear View

Introducing ASR9906

BRKARC-2003 10


Cisco ASR 9910 Overview

Feature Description

Capacity 64 Tbps; mid-plane architecture

Capacity /Slot 4 Tbps


Rack size 21RU

Power 2 Power Trays (4.4 KW DC supplies; 6.0 KW AC supplies)

Fan Front-to-Back Airflow. 2 Fan Trays, FRU


Fabric Cards

5 Fabric cards in the rear for additional capacity

230G per FC

Up to 6+1 Redundancy using RSP’s integrated fabric

Line cards

Fixed high density 10/100GE Typhoon and Tomahawk LCs

Modular

MOD80, MOD160, MOD200, MOD400

MPAs: 20xGE, 4x10G, 8x10G, 2x40G, 1x100G, 20x10G, 2x100G

Low Density Fixed

40x1GE, 4x10G+16x1GE

Service Card: Virtualized Service Module (VSM)

5 Fabric Cards

2 Fan Trays

2 RSPs

8 LC Slots

2 Power Trays

BRKARC-2003 11


ASR 9000 Linecards Evolution

3rd Gen

Tomahawk

Class

800G

Tomahawk

28nm

240 Gbps

Tigershark

28nm

200 Gbps

SM15

28nm

1.20 Tbps

X86

6 Core

2 Ghz

2nd Gen

Typhoon

Class

360G

Typhoon

55nm

60 Gbps

Skytrain

65nm

60 Gbps

Sacramento

65nm

220 Gbps

PowerPC

Quad Core

1.5 Ghz

Trident

90nm

15 Gbps

Octopus

130nm

60 Gbps

Santa Cruz

130nm

90 Gbps

PowerPC

Dual Core

1.2 Ghz

1st Gen

Trident

Class

120G

BRKARC-2003 12


ASR 9K Ethernet Line Card Overview

A9K-MOD80 A9K-MOD160

MPAs20x1GE2x10GE4x10GE1x40GE2x40GE

-TR, -SE

MPAs1x100GE2x100GE20x10GE

+ Typhoon MPAs

MOD400/MOD200A9K-4x100GE A9K-12x100GE

A9K-36x10GE A9K-24x10GE

A9K-40GE

A9K-4T16GE

2nd LC Typhoon

NPU: 60Gbps,

~45Mpps

3rd LC Tomahawk

NPU: 240Gbps,

~150Mpps

A9K-8x100GE A9K-400G-DWDM24/48x10/1G LC

BRKARC-2003 13


Tomahawk Forwarding Engine and LC

• High Density/Scale

• 240 Gbps throughput/ 150 MPPS+ for “line

rate” 100GE ports

• Increased scale for multi-dimensional

profiles

• 2M MAC addresses(6M Capacity)

• 10M v4 or 5M v6 routes

• 1M queues/NP

• Flexible Interface Support

• Supports dense 100GE/40GE/10GE

• “Mix and match” interface types on the same

LC

• CPAK Advantage

• Pluggable CPAK based on customer

demand (1x100GE, 2x40GE, 10x10GE)

• In-line Security

• MACSec support in-line on LC at line rate

100G (800G/LC)

• “Green” Design

• Ability to power down components for power

savings

• High Availability Customized Silicon – Hitless

FPD Upgrades

• Hardware Offload of CPU Intensive Protocols

• Elimination of loopback for most feature paths

BRKARC-2003 14


Tomahawk LC: 8x100GE Architecture

XB

AR

Se

pa

rate

d S

witc

h F

ab

ric

LC CPU

TomahawkNP

FIAPHY

TomahawkNP

FIAPHY

TomahawkNP

FIAPHY

TomahawkNP

TigersharkFIA

PHY

CPAK:

100G, 40G, 10G

L2/L3/L4 lookups, all VPN types, all

feature processing, mcast

replication, QoS/Queuing, ACL, etc …

Slice Based Architecture

Macsec Suite B+, G.709, OTN,

Clocking

VoQ buffering, Fabric credits,

mcast hashing, scheduler for fabric

and egress port

240G

240G

240G

240G

240G

240G

240G

240G

FPOE, Auto-Spread,

DWRR, RBH, replication

Per Slice Power Management: (100-200W Power Savings)

PE1(admin-config)# hw-module power saving location ? slice [0-3]

Up to

14x115G

BRKARC-2003 15


Line Card Components

NPUFIA Switch

Fabric ASIC

TM

PHY

CPU

P1

P2

P3

BEP1

P2

P3

BE

Runs distributed control plane protocols for increased scale

BFD, CFM, ARP

Receive FIB table from RP and program hardware forwarding table

Provides data connection to switch fabric

Manage VoQ, Superframe and loadbalancingdata traffic across switch fabric

Mcast replication table for replication toward NPs

Main forwarding engine L2 and L3 lookups

Multicast replication toward Optics

User level QoS and Security features

Dedicated queue ASIC – TM (traffic

manager) per NPU for QoS functions

User Configurable Queue on TM.

Default Port Queue Always Created.

BRKARC-2003 16


Network Processor Architecture Details

• TCAM: VLAN tag, QoS and ACL classification

• Stats memory: interface statistics, forwarding statistics etc

• Frame memory: buffer, Queues

• Lookup Memory: forwarding tables, FIB, MAC, ADJ

• TR/SE

• Different TCAM/frame/stats memory size for different per-LC QoS, ACL, logical interface scale

• Same lookup memory for same system wide scale mixing different variation of LCs doesn’t impact system wide scale

-

STATS MEMORY

FRAME MEMORYLOOKUP

MEMORY TCAM

FIB MAC

NPU Complex

Forwarding chip (multi core)

TR and SE has different

memory sizeTR and SE has same

memory size

-TR: transport optimized, -SE: Service edge optimized

BRKARC-2003 17


Tomahawk Line Card Architectures

4x100G/40x10G Port LC

MOD400 LC

Switch

Fabric

CPU

Up to

14x115GSwitch

Fabric

FIA

FIA

FIA

FIANP

NP

NP

NP

PHY

PHY

PHY

PHY

CPAK

CPAK

CPAK

CPAK

CPU

Switch

Fabric

FIA

FIANP

NP

SerDes

XBAR

Bay

Bay

CPU

P

H

Y

P

H

Y

MOD200 LC

Up to

14x115G

Switch

FabricFIANPSerDes

XBAR

Bay

Bay

CPU

P

H

Y

P

H

Y

Up to

14x115G

Up to

14x115G

FIAPHY NP

FIAPHY NP

FIAPHY NP

FIAPHY NP

FIAPHY NP

FIAPHY NP

12x100G Port LC

BRKARC-2003 18


• Supports 1G, 10G HW configurations available on Typhoon

• Feature and scale parity on PAR with Typhoon LC with minimal exceptions

• No OTN/MACSec• Support for all SFP+/SFP currently

available on Typhoon• 64bit XR capable

Ultra-Dense dual rate 1G/10G fixed line card.

Available in 24-port and 48-port versions

(Oversubscribed)

Powered by the latest generation Tomahawk NPU

& ASICS

Price and power optimized LC

• Ideal low cost replacement for both fixed and modular Typhoon LC

• PAYG consumption model available

Dual-rate dense 10G/1G Tomahawk LCA9K-48x10G-1G – Enables easy migration from Typhoon

BRKARC-2003 19


• Supports all 1G and 10G. 1G and 10G are configurable follow certain rule

• 48-port has a two NPU slices with shared 80Mb TCAM

• PHY does not support OTN Framing or MACSec

Ports0-7 PHY0

FIA SM15Ports8-15 PHY1

Ports16-23 PHY2

NPU 0

SFPs0-7

SFPs8-15

SFPs16-23

IFE

0IF

E1

Ports24-31 PHY3


Ports40-47 PHY5

NPU 1

SFPs24-31

SFPs32-39

SFPs40-47

IFE

0IF

E1

• 240G Raw Bandwidth/ 200G

Data Traffic Bandwidth

• Potentially Oversubscribed

8 Ports

8 Ports

4 Ports

4 Ports

A9K-48x10G-1G Architecture

BRKARC-2003 20


• Supports all 1G and 10G. 1G and 10G are configurable follow certain rule

• 24-port has a single NPU slice with 40Mb TCAM

• Phy does not support OTN Framing or MACSec

Ports0-7 PHY0


Ports16-23 PHY2

NPU 0

SFPs0-7

SFPs8-15

SFPs16-23

IFE

0IF

E1

• 240G Raw Bandwidth/ 200G

Data Traffic Bandwidth

• Potentially Oversubscribed

8 Ports

8 Ports

4 Ports

4 Ports

A9K-24x10G-1G Architecture

BRKARC-2003 21


A9K-48x10G-1G Port Organization• A9K-48X10GE-1G-SE/TR has two NPU slices, A9K-24X10GE-1G-SE/TR has one

• Each NPU slice consists of two Interface Engines (IFE)

• Interface Ports are organized in port-groups

• Each port-group consists of 4 interface ports

• Each IFE consists of three port-groups

A9K-48X10GE-1G-SE/TR

00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

TOMAHAWK NPU

IFE

TOMAHAWK NPU

IFE IFE IFE

12 ports per IFE 12 ports per IFE 12 ports per IFE 12 ports per IFE

port-group port-group port-group port-group port-group port-group port-group port-group port-group port-group port-group port-group

BRKARC-2003 22


A9K-48x10-1G & A9K-48x10-1G Port Speed Rules

• Interface Port Speed is configurable: 1G / 10G

• Per default, all ports are 10G

• The 4 Rules:

1. All ports in a port-group (group of four ports) have to have the same speed.

2. If the first port-group on an IFE (group of three port-groups) is configured as 1G, then all ports in that same IFE must be 1G.

3. If the first port-group of an IFE is 10G, then the other two port-groups in that same IFE can be any combination of 1G or 10G.

4. When configuring port speeds, all ports of a given linecard need to be configured together in one single CLI command.

• Deviations from these rules will results in a CLI rejection

• Changing port speed on an interface does not impact traffic on other interfaces

BRKARC-2003 23


Possible Port Speed Combination per IFE

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11

Configuration 1 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G 1G





Interface Engine (IFE)

• 5 possible port speed combinations for each IFE

• IFE are independent of each other for port speed configuration

BRKARC-2003 24


Port Speed Configuration Example

hw-module location <rack>/<slot>/CPU0 port-mode RxS,RxS[,RxS…]

R = Run-length (number of consecutive ports with same speed)

S = Speed (1 or 10, in Gbps)

EXAMPLE:

hw-module location 0/4/CPU0 port-mode 8x10,4x1,4x10,8x1,20x10,4x1

00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

BRKARC-2003 25


• MPA’s Supported:2x100GE CPAK. 1x100GE CPAK. 20x10GE SFP+All Typhoon MPA

• Flexibility to use CPAK 10G/40G/100G optics on 100G MPAs

• 2 Flavors for FlexibilityMOD400 has 2 Tomahawk ASICSMOD200 has 1 Tomahawk ASIC

MPA #1(2x100G)

MPA #2(20x10G)

Mo

d4

00

LC

2 x 100GCPAKs

20 x 10G SFP+

MOD200Support Matrix

Comb 1 Comb 2 Comb 3

EP0 2x100G-MPA 20x10G-MPA

1x100G or anyTyphoon MPAs

EP1 None None 1x100G or any Typhoon MPAs

MOD400Support Matrix

Comb 1 Comb 2 Combo 3 Comb 4

EP0 2x100G-MPA 20x10G-MPA 2x100G-MPA 1x100G or anyTyphoon MPAs

EP1 2x100G-MPA 20x10G-MPA 20x10G-MPA 2x100G-MPA OR20x10G-MPA

26BRKARC-2003

Tomahawk Modular LCs – Mod400 & Mod200


CPAK Interface Flexibility

Configurable 100GE Interconnect

Options for 10GE interfaces:

10GE Interconnect Options

40GE Interconnect Options

hw-module 0/x/cpu0 port z breakout 10xTenGigE

hw-module 0/x/cpu0 port z breakout 2xFortyGigE

DUPLEX SC TO

LC /SC/ST SM

CPAK-100G-LR4

LGX

Panel

MPO24 TO 10X

DUPLEX

LC/SC/ST MM

CPAK-100G-SR10

CPAK-10X10G-LRMPO24 TO 10X

DUPLEX LC

/SC/ST SM

LGX

Panel

LGX

Panel

CPAK-2X40G-LR4

LC TO DUPLEX

LC/SC/ST SM

LGX

Panel

Interface HunGigE 0/x/y/z

Breakout Interface Convention:

Rack/Slot/Bay/Port (phy)/Breakout#

Interface TenGigE 0/x/y/z/0


…


Interface FortyGigE 0/x/y/z/0

Interface FortyGigE 0/x/y/z/1

BRKARC-2003 27

ASR 9000 Distributed Control Plane


ASR 9000 System Architecture “At-a-Glance”

Fully Distributed Architecture for

High Performance and High Multi-

dimensional Control Plane Scale

Active-Active Switch Fabric

RSP

RSP/RP

FIC

CPU

BITS/DTI

FIA

CPU

Switch

Fabric

FIA

FIA

FIA

FIANP

NP

NP

NP

PHY

PHY

PHY

PHY

CPAK

CPAK

CPAK

CPAK

CPAK

CPAK

CPAK

CPAK

CPU

CPU

Switch

Fabric

FIA

FIA NP

NP

SerDes

XBAR

Bay

Bay

CPU

Switch Fabric

Line Card

Data forwarding is fully

distributed across NPs

Control plane split

among RSP/RP

and LC CPU

Network Processor

Network Processor

BRKARC-2003 29


Switch

Fabric

FIA

FIA

FIA

FIANP

NP

NP

NP

PHY

PHY

PHY

PHY

CPAK

CPAK

CPAK

CPAK

CPAK

CPAK

CPAK

CPAK

CPU

ASR9000 Fully Distributed Control Plane

Switch Fabric

PuntFPGA FIA

Switch Fabric

RP

LPTS

LC CPU: ARP, ICMP, BFD, NetFlow,

OAM/CFM, L2 Protocols etc

RP CPU: Routing, MPLS, IGMP, PIM,

HSRP/VRRP, etc

LPTS (local packet transport service):

control plane policing

Control packet

Punt Switch

CPU

CPU

NP Offloading: BFD

LC

BRKARC-2003 30


LC NP

31BRKARC-2003

L3 Control Plane Architecture

LC CPU

RSP/RP CPU

LDP RSVP-TE BGP

ISIS

OSPF

EIGRP

Static

HW FIB Adjacency

ARP/NDP

LSD RIB

AIB

SW FIB

AIB: Adjacency Information Base

RIB: Routing Information Base

FIB: Forwarding Information Base

LSD: Label Switch Database

RSP/RP

LC

ISIS/OSPF BGP


Distributed ARP Processing

LC

CPU

PHY NP FIALPTS

Punt Switch

ARP

spio netio

Tsec Driver

SPP

Line card

FIA

Sw

itch

Fab

ric

RP

Sw

itch

Fab

ric

ARP

RP CPUIncomplete ADJ Packets

Incoming ARP Packets

Outgoing ARP Packets

BRKARC-2003 32


NP

Switch Fabric

3x10GE

SFP +

3x10GE

SFP +

3x10GE

SFP +

3x10GE

SFP +

3x10GE

SFP +

3x10GE

SFP +

3x10GE

SFP +

3x10GE

SFP +

FIA

FIA

FIA

FIA

Sw

itch

Fab

ric A

SIC

CPU

PuntFPGA FIA

CPU

Switch Fabric

RP

LC1

3x10GE

SFP +

3x10GE

SFP +

NP

NP

3x10GE

SFP +

3x10GE

SFP +

NP

NP

3x10GE

SFP +

3x10GE

SFP +

NP

NP

3x10GE

SFP +NP

NPFIA

FIA

FIA

FIA

Sw

itch

Fab

ric A

SIC

CPULC2

1 2

2

NP learn MAC address in hardware (around

4M pps)

NP flood MAC notification (data plane)

message to all other NPs in the system to sync

up the MAC address system-wide. MAC

notification and MAC sync are all done in

hardware

1

2

NP

NP

NP

NP

NP

NP

NP

Hardware based MAC learning: ~4Mpps/NP

33BRKARC-2003

MAC Learning and Sync

Data packet


Distributed BFD Architecture

OSPF ISIS BGP

BFD Session Tables on RP

BFD Events

BFD session info

RP

BFD Session Tables on LC

BFD EventsBFD Session info

LC1-CPU

BFD Hellos

BFD Session Tables on LC

BFD Hellos

BFD Session info

……

LCn-CPU

BRKARC-2003 34


HW-offloaded BFD

OSPF ISIS BGP

BFD Session Tables on RP

BFD Events

BFD session info

RP

BFD EventsBFD Session info BFD Session info

……

BFDOnNP

LCn-CPU

BFD Hellos

BFDOnNP

BFDOnNP

BFDOnNP

BFDOnNP

LC1-CPU

BFD Hellos

BFDOnNP

BFDOnNP

BFDOnNP

hw-module bfd-hw-offload enable location 0/0/CPU0

BRKARC-2003 35


Distributed Netflow Architecture

NPU

LC-CPU1M Record Cache

NPU NPU NPU

RPNetflow show/clear

Command

Netflow

Configure

NPU: • Traffic Filtering

• Traffic Sampling

• Extract flow

header information

• LPTS Policer

• Aggregate 200Kpps netflow

policer rate

• Evenly Divided among

netflow-enabled NPs

netIO

EXPORT

• Manage flow table

• Export flow

information

LC

BRKARC-2003 36

ASR 9000 Data Packet Processing


CPAK 0

NP FIA

FIA

FIA

FIA

CPU

CPAK 1

PHY

CPAK 2

NP CPAK 3

PHY

CPAK 4

NP CPAK 5

PHY

CPAK 6

NP CPAK 7

PHY

…

Up to

14x120

G

Switch Fabric (SM15)

CPAK 0

NP FIA

FIA

FIA

FIA

CPU

CPAK 1

PHY

CPAK 2

NP CPAK 3

PHY

CPAK 4

NP CPAK 5

PHY

CPAK 6

NP CPAK 7

PHY …

Up to

14x120G

Switch Fabric (SM15)

Switch Fabric

Switch Fabric

Distributed Two-Stage Packet Processing

• Ingress lookup yields packet egress port and applies ingress features

• Egress lookup performs packet-rewrite and applies egress features

Uniform packet flow for simplicity and predictable performance

12

1

2

BRKARC-2003 38


ASR 9000 Life of a Packet – Tomahawk LC

• MACSEC Decryption• G.709/OTN/WAN-

PHY/LAN-PHY• Line Clocking

PHY

FIA

CPUCPU

Switch Fabric

CPAK 0

NP FIACPAK 1

PHY

TM

12

3

4

5

NPTM

6

7

8

1

• Ingress L2/L3 FIB lookup, ACL/QoS lookup

• Ingress PBR/ABR, ACL, uRPF• Ingress QoS: classification,

marking, policing• Packet Punting• Ingress ECMP/LAG hashing

2

Ingress side of LC

Egress side of LC

• Ingress Queuing Processing

• Bypassed in case no ingress queuing support

3

• Buffering packet from NP• Requesting fabric credit• Manage superframe and

load-balancing packet across fabric

• Manage system VoQ

4

• Re-assembling packets from superframe

• Send packet to corresponding NP

• Release buffer and fabric credit

5

• Egress L2/L3 FIB lookup, ACL/QoS lookup

• Egress PBR/ABR, ACL, uRPF• Egress QoS: classification,

marking, policing, shaping• Incomplete Adj Packet Punting• Egress ECMP/LAG hashing

6

• MACSEC Encryption• G.709/OTN/WAN-

PHY/LAN-PHY• Line Clocking

8

• Egress Queuing Processing

7

BRKARC-2003 39


ASR 9000 Switch Fabric High-Level Architecture3-Stage Non-Blocking Fabric (Separate Unicast and Multicast Crossbars)

FIAFIA

FIARSP0

Arbiter

fabric

RSP1

Arbiter

fabric

fabricFIAFIA

FIA

fabric

Fabric frame format:

Super-frame

Fabric load balancing:

Unicast is per-packet

Multicast is per-flow

Virtual Output Queue

Arbitration

Stage 1 Stage 2 Stage 3

TomahawkTyphoon LC

Ingress LinecardEgress Linecard

Active-Active Fabric

BRKARC-2003 40


Upto 1.2T linerate/LC capacity

7 Fabric: 6+1 redundancy

800G linerate /LC (protected) capacity – all

Typhoon, Tomahawk LCs (except

Skyhammer) interoperate at full throughput

SFC2

SM15

5-Fabric LC

1.2T with 6 Fabric

1.4T with 7 Fabric

7-Fabric LC

SM15

200G

1T with 5 Fabrics

800G w/ 4 Fabrics

200G

ASR 99xx Switch Fabric – SFC2 and Mixed Gen LCs

100G 83G100G

BRKARC-2003 41


Switch Fabric Arbitration

Fabric ASIC

and VOQ

SFC or RSP1

Arbitration

Fabric

ASIC

Fabric

ASIC

Fabric

ASIC

Fabric

ASIC

ArbitrationFabric ASIC

and VOQ SFC or RSP0

1: Fabric Request

3: Fabric Grant

2: Arbitration

4: load-balanced transmission across fabric links

5: credit return


• Unicast traffic sent across first available fabric link to destination (maximizes efficiency)

• Each frame (or super frame) contains sequencing information

• All destination fabric ASIC have re-sequencing logic

• Additional re-sequencing latency is measured in nanoseconds

Fabric Load Balancing – Unicast

Fabric ASIC

and VOQ

SFC or RSP1

Arbitration

Fabric

ASIC

Fabric

ASIC

Fabric

ASIC

Fabric

ASIC


and VOQ SFC or RSP04 3 2 1

BRKARC-2003 43


RSP880

RSP880

Tomahawk Line Card

FIA

Fabric

SM15

FIA

FIA

FIA

NPSFP+SFP+SFP+

SFP+SFP+

NP

NP

NP

SFP+SFP+SFP+

SFP+SFP+

SFP+SFP+SFP+

SFP+SFP+

SFP+SFP+SFP+

SFP+SFP+

Distributed Multi-Stage Multicast Replication

CPAK 0

NP FIA

FIA

FIA

FIA

CPAK 1

PHY

CPAK 2

NPCPAK 3

PHY

CPAK 4

NPCPAK 5

PHY

CPAK 6

NPCPAK 7

PHY

Up to

14x120

G

Switch Fabric(SM15)

Tomahawk Line Card

FIA

Fabric

SM15

FIA

FIA

FIA

NPSFP+SFP+SFP+

SFP+SFP+

NP

NP

NP

SFP+SFP+SFP+

SFP+SFP+

SFP+SFP+SFP+

SFP+SFP+

SFP+SFP+SFP+

SFP+SFP+

1

1. 512k MGID/FGID lookup

Flow based (RBH key calc)

multi link load balance

2

3

2 & 3. Flow based (using RBH)

multi link load balance

4

4. 22-bit FGID for slot bitmask

based unique fabric LC

replications and FPOE lookup

5

5

5. MGID lookup into FPOE

table (256k entries) for

unique FIA replications

6

6

6

6

6. MGID in FIA DI table lookup

for unique NPU replication

Flow based load balance

7

7

7. MGID lookup for NPU

to egress OIF interface

replication

7

7

BRKARC-2003 44


• Multicast traffic hashed based on (S,G) info to maintain flow integrity

• Very large set of multicast destinations preclude re-sequencing

• Multicast traffic is non arbitrated – sent across a different fabric plane

Fabric Load Balancing – Multicast

Fabric ASIC

and VOQ

SFC or RSP1

Arbitration

Fabric

ASIC

Fabric

ASIC

Fabric

ASIC

Fabric

ASIC


and VOQ SFC or RSP0A

1

A

2

B

1

A

3

B

2

C

1

Flows exit in-order

BRKARC-2003 45

ASR 9000 QoSArchitecture


ASR 9000 Priority-Based QoS Architecture

• Ingress (sub-)interface QoS Queues

• User Configurable Ingress QoS Policy

• 4xVOQ per VQI• Up to 8K VOQs per TSK FIA

(vs 4k per SKT FIA)

4x Egress Destination Qs per VQI, aggregated at egress port rate

• Dedicated Traffic Manager(TM) for Traffic Queuing

• User Configurable QoS Policy on Ingress/Egress NP

• End-to-End priority propagation Guarantee bandwidth, low latency for high priority traffic

• Unicast VOQ and back pressure

User-configuration with Ingress MQC

Egress (sub-)interface QoS Queues

User-configuration with Egress MQC

Implicit ConfigurationNot User-controllable

Ingress side of LC Egress side of LC

NP0 PHY

NP2 PHY

NP3 PHY

NP1 PHYFIA

CPUCPU

Switch Fabric

3

4

1 2 34

CPAK 0

NP FIACPAK 1

PHY

1

2

P1

P2

P3

BE

P1

P2

P3

BE

P1

P2

P3

BETM

TM

P1

P2

P3

BE

BRKARC-2003 47


Default Implicit Trust Model

internal

cos = 1L2 Bridging

802.1p = 1 *

802.1p = 1IPP=5 DSCP=44

IPP=5 DSCP=44

IPP=5 DSCP=44

internal

cos = 0L2 Bridging

UntaggedIPP=5 DSCP=44

802.1p = 0 *IPP=5 DSCP=44

IPP=5 DSCP=44

Ingress line card Egress line card

Carried in internal buffer header,

by default, internal cos is used for impositioned fields only,

For example, added vlan tag, impositioned MPLS label,

It doesn’t include VLAN tag translation or MPLS label swap

ASR 9000 would never modify

packet DSCP/IP without

a policy-map configured

L2 IF: trust outer Cos

L3 IF: trust DSCP

L3 MPLS: trust outer EXP

BRKARC-2003 48


Define Traffic ClassConfigure Class-Map to Identify Different Classes of Traffic

Match on Different L2/L3 and System Internal Parameters

Associate Policies/Action to Each Traffic ClassConfigure Policy-Map

Policer, Shaper, Queuing, Bandwidth, Marking/Remarking

Always class-default to Match all No-Match Traffic

Attach Policy to TargetL2/L3 Physical Main Interface/Sub-interface

Bundle-Ether, BVI, PWHE Main Interface/Sub-interface

Both Ingress and Egress Directions

Class-map:

Classification

Policy-map:Action to Class

Policy Attachment

MQC – Three-Step QoS Configuration Model

BRKARC-2003 49


ASR 9000 QoS Classification Criteria

L2 Header Fields L3 Header Fields Internal Marking

L2

Interfaces/EF

Ps

Or L3

Interfaces

Inner/outer COS,

inner/outer vlan,

DEISource/Destination MAC address*Ethernet Access-List

Outer EXPDSCP/TOSTTL, TCP flags, Source/destination L4 portsProtocolSource/Destination IPv4address*Access-List

Flow-key/CAC

Discard-classQos-groupTraffic-class

Notes: - Support match all or match any

- Max 8 match statements per class, max 8 match entries per match statement

- Not all header fields can be used in one MQC policy-map, see details next

BRKARC-2003 50


•Priority Level 1•Priority Level 2•Shape, •bandwidth or bandwidth remaining•(W)RED•Police•Set (marking)

•Priority Level 1•Priority Level 2•Shape, •bandwidth or bandwidth remaining•(W)RED•Police•Set (marking)

•Shape

•Bandwidth

remaining

•1R2C policer

L4

Child policy (child of

Parent) with user-

defined classes &

class-default

L3Parent Policy on L2 (EFP) or L3 subint. class-

default or physical port /w user-

defined classes

L2Grand-Parent

Policy

L1

Port

scheduler

not

configurable

N/A

policy-map grand-parent (configures L2 layer)

class-default

shape average x

service-policy parent

policy-map parent (configures L3 layer)

class c1

priority level 1

police rate y mbps

class c2

class-default

service-policy child

policy-map child (configures L4 layer)

class-1

priority level 2

class-2

bandwidth remaining percent x

class class-default

int GigE 0/1/2/3.10 l2transport

encapsulation dot1q 10

service-policy output parent

3-Layer Hierarchical QoS (H-QoS)

L0

Port Group

scheduler

not

configurable

N/A

BRKARC-2003 51


H-QoS – Supported Classification/Policy

Policy-map hierarchy level Classification support Policy Support

Grand-parent Only class-default

• Shape Average

• Bandwidth remaining

• 1R2C policer with only drop/transimit

action(no set/mark, Tomahawk card only)

Parent

User defined fields with

restrictions based on

format/interface types.

• Priority/WRED Queue and Queue-limit on

Leaf only

• Policer/Shaper/Marking/non-Priority

Queue/Bandwidth/Bandwidth Remaining

Child

User defined fields with

restrictions based on

format/interface types.

• Priority/WRED Queue and Queue-limit on

Leaf only

• Policer/Shaper/Marking/non-Priority

Queue/Bandwidth/Bandwidth Remaining


Default Interface Queues

P1 P2 P3 L

Level 4

Level 3

Level 2

Level 1

Queues

Schedulers

Schedulers

Schedulers

BRKARC-2003 53


MQC Hierarchy in Queuing ASIC

policy-map child

class c1

priority level 1

police rate 640 kbps

class c2

bandwidth 20 mbps

class class-default

bandwidth 1 mbps

!

policy-map parent

class class-default

shape average 35 mbps


!

interface GigabitEthernet0/0/0/0


P1 P2 P3 L

Port default queues

c1 c2 cd-c

MQC queues

Inactive entity

Active entity

cd-p

L1

L2

L3

L4

cd-c

cd-p


MQC Hierarchy in Queuing ASICpolicy-map child

class c1

priority level 1


class c2

bandwidth 20 mbps

class class-default

bandwidth 1 mbps

!

policy-map parent

class class-default



!

interface GigabitEthernet0/0/0/0.1


!



Port default queues

c1 c2 cd-c

G0/0/0/0.1

Inactive entity

Active entity

P1 P2 P3 L c1 c2 cd-c

G0/0/0/0.2

cd-p

L1

L2

L3

L4

cd-p

cd-c

cd-p


MQC Hierarchy in Queuing ASIC

policy-map child

class c1

priority level 1


class c2

bandwidth 20 mbps

class class-default

bandwidth 1 mbps

!

policy-map parent

class c3



class class-default

bandwidth 1 mbps

!

policy-map grand-parent

class class-default


service-policy parent

!


service-policy output grand-parent

Port default queues

c1 c2 cd-c

G0/0/0/0.1

P1 P2 P3 L cd-p-vc

c3

L1

L2

L3

L4

cd-p

cd-c

cd-p

cd-gpcd-gp

IOS-XR & IOS-XR 64 Bit


IOSd

Kernel

Linux-BinOSH

oste

d A

pp

1

Hoste

d A

pp

2

Operational Infra

Virtualization Layer

IOS “Blob”

Cisco IOS Cisco IOS-XE Classic IOS-XR IOS-XR 64 Bit

XR Code v2

Kernel

Linux, 64bit

Distributed Infra

BG

P

OS

PF

PIM

AC

L

QoS

LP

TS

SN

MP

XM

L

NetF

low

Kernel

Linux, 64bit

System

Admin

XR Code v1

Kernel

Linux, 64bit

Distributed Infra

BG

P

OS

PF

PIM

AC

L

QoS

LP

TS

SN

MP

XM

L

NetF

low

Kernel

QNX, 32bit

Distributed Infra

BG

P

OS

PF

PIM

AC

L

QoS

LP

TS

SN

MP

XM

L

NetF

low

Control

Plane

Data

Plane

Mgmt

Plane

1990s 2000s 2003-04 Present Day

Cisco IOS – A Recap

Incremental Development, with Industry leading investment protection

BRKARC-2003 58


IOS XR Evolution: XR 64 Bit Architecture

IOS-XR

IOS XR

QNX

QNX

Linux

Linux

Routing Apps

SystemAdmin

Routing Control Plane

Admin Plane

LC-CPUs

RP

Lin

e C

ard

LC-CPU

Separate Admin Plane

64 bit Linux Kernel

Linux VM

64-bit IOS XR.

Admin Plane

Linux-based Virtualized

IOS XR 64 Bit

RP

Lin

e C

ard

IOS-XR

IOS-XR

Linux Linux

Linux Linux

Classic IOS XR


IOS XR 64 Bit Internal Connection


Admin Plane

LC-CPUs

Admin Plane

RP

1

Lin

e C

ard

IOS-XR

IOS-XR

Guest OS Guest OS

Guest OS Guest OS

Host OS


Admin Plane

RP

2

IOS-XR

Guest OS Guest OS

Host OS

Host OS

LC-CPUs

Admin Plane

IOS-XR

Guest OS Guest OS

Host OS

10.0.2.0/24 10.0.2.0/24

10.0.2.0/24 10.0.2.0/24

192.0.0.0/8


IOS XR 64 Bit CLI Access

Linux

Linux

Admin Plane

LC-CPUs

Admin Plane

RP

Line Card

IOS-XR

IOS-XR

Linux Linux

Linux Linux

Admin CLI

Admin Linux CLI

Host Linux CLI

XR CLI

XR Linux CLI

run

admin

run

ssh my_host

Console

Mgmt

• Level of Access Controlled by User Authority

• Linux CLI: Only root-lr/root-system users

• ”exit” to return to previous level


Operations in Different Access CLI—on XR VM

Start from IOS XR CLI

“run” to access XR Linux CLI

Linux CLI Execution

“exit” to return to previous CLI


Operations in Different Access CLI—on admin VM


“admin” to access admin CLI

“run” to access admin Linux CLI

Linux CLI Execution

Predefined hostname in /etc/hosts file.



Operations in Different Access CLI—Host Linux CLI


“admin” to access admin CLI

“run” to access admin Linux CLI

“ssh my_host” to host Linux CLI

Linux CLI Execution

”virsh list” to show two VMs:

Admin and XR VMs



Default Management Interface Mapping


Admin Plane

RP

1

IOS-XR

Guest OS Guest OS RP

2 Routing Control Plane

Admin Plane

IOS-XR

Guest OS Guest OS

IOS-XR

QNX

Routing Apps

SystemAdminR

P1

Classic IOS XR

IOS-XR

QNX

Routing Apps

SystemAdminR

P2

IOS XR 64 Bit


How to Copy Files from/to Admin VM

BRKARC-2003 66

Admin Plane IOS-XR

Linux Linux

TFTP

x

No admin MGMT connection

Admin Plane IOS-XR

Linux Linux

TFTP

With admin MGMT connection

1

2

1. In admin linux CLI, “scp $file $act_RSP_XR:/$dir”. Replace act_RSP_XRwith the actual active RSP XR hostname

2. In XR exec CLI, use the copy command. TFTP/FTP/HTTP options supported

1

1. In admin CLI, use the copy command. Only TFTP option.


Same XR Control Plane: XR Feature Configurations Stay Unchanged

Different Boot Process

Classic XR: ROMMON. CLI based. Support TFTP/USB boot. No ZTP capability

XR 64 Bit: Open-source iPXE. Menu/Scripts process control.IPv4/IPv6 support. TFTP/FTP/HTTP/USB boot. ZTP capable. Directly boot from ISO instead of Turboboot.

Different CLI to Reload Chassis

Classic XR: “Reload location all” from ADMIN exec prompt to reload whole chassis

XR 64 Bit: “hw-module location all reload” or “reload rack 0” from ADMIN exec prompt to reload the whole chassis. “reload location all” reload the VMs only.

XR 64 Bit image upgrade: Admin VM XR VM

XR 64 Bit: Added Install commands (update/upgrade) for flexible package management

XR 64 Bit: Added “install activate issu” to support ISSU operation

Admin CLI changes: Configuration, Exec and Show commands

XR Exec and Show command CLI change. No major XR configuration CLI change

XR 64 Bit Operational Differences


IOS XR 64 Bit Packaging

Bootable Images

Minimum Image asr9k-mini-x64.iso-6.1.2.iso Core packages: OS, Admin, Forwarding, Modular

Services Card, Basic Routing, SNMP, Alarm Correlation

asr9k-minik9-x64.iso-6.1.2.iso Core packages + Encryp/decryp, SSL/SSH/PKI

Golden ISO Customized ISO image including mini ISO+required packages and SMUs

Optional Feature Packages

asr9k-eigrp-x64-1.0.0.0-r612.x86_64.rpm

asr9k-isis-x64-1.1.0.0-r612.x86_64.rpm

asr9k-ospf-x64-1.1.0.0-r612.x86_64.rpm

asr9k-m2m-x64-2.0.0.0-r612.x86_64.rpm

asr9k-mgbl-x64-3.0.0.0-r612.x86_64.rpm

asr9k-mpls-te-rsvp-x64-1.2.0.0-r612.x86_64.rpm

asr9k-mpls-x64-2.1.0.0-r612.x86_64.rpm

asr9k-mcast-x64-2.0.0.0-r612.x86_64.rpm

asr9k-optic-x64-1.0.0.0-r612.x86_64.rpm

asr9k-li-x64-1.1.0.0-r612.x86_64.rpm

asr9k-k9sec-x64-3.1.0.0-r612.x86_64.rpm

BRKARC-2003 68


Migrating Classic XR to IOS XR 64 Bit

FPD Upgrade

Pre-6.1.2

5.x.y

4.x.y

Post-6.1.2

6.2.x

6.3.x

Two Ways to Migrate:

• CSM Orchestrated Migration

• MOP: Manual Migration

• Follow below link for details

XR 6.1.2

or later

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/migration/guide/b-migration-to-ios-xr-64-bit/b-migration-to-ios-xr-64-bit_chapter_00.html


CSM Environment For Image Management

CSM

Image Repository

CCOLocal

Server

TFTP/

SFTP/

FTP

• CSM Provides Web-GUI Interface

• Manage Software Packages for

Group of Network Devices

• Local Packages Repository or CCO

• TFTP/SFTP/FTP Needed

• Files can be pre-copied on router

• Follow the link for CSM details

http://www.cisco.com/c/en/us/td/docs/routers/asr90

00/software/smu/csmuser.html


IOS XR 64 Bit -- Technical Benefits

Virtualized Architecture

ISSU Ready

MPL, ZTL

Reduced Upgrading Time

High Scale

e.g. 10M IPv4/5M IPv6

Open Linux-kernel Baesd

Linux RPM Packaging

Offline Package Management

Single Bootable ISO Package

APP Hosting

Open and Secure

Multi-Level API

iPXE

Zero-Touch Provisioning

IOS XR 64 Bit

Telemetry

Both XR 64 Bit & Classic XR

BRKARC-2003 71


System Architectural Support for ISSUDifferent Version IOS XR Simultaneously Running

V1

Slice 2

LC (ZPL/MPL)RP (ZTL/MTL)

Ad

min IOS

XR V1

Linux Slice 1 Slice 3 Slice 4

V2

Ad

min IOS XR

V1

IOS XR

V2

Linux

LC-CPU

Tomahawk only

Ad

min IOS

XR V2

Linux

Active StandbyStandby Active


Application Hosting

• Device Extension with off the shelf applications

• Multi-level API access

• Native Linux APIs

• ASIC SDK level APIs

• OS level APIs

• Complete tool chain integration

• Easy to use and integrate with customer build

environments

Hardware

Linux

Cisco OS

Linux Based

Applications

Apps Built for

Cisco OS

FWD ASIC SDK

Ap

ps A

pps

Ap

ps

Ap

ps

BRKARC-2003 73

App Hosting


App Hosting: Modes of Hosted Applications

HOST OS/HYPERVISOR

Ad

min

Pla

ne

VM

XR VM APP Process

in XR VMAPP in Container

Linux Guest OS

XR Lib/Pkgs

App Lib/Pkgs App Lib/Pkgs

LXC/Docker

Container

RP CPU

Linux

App Hosting


Conclusion

• ASR 9000 - Truly Carrier-Class Edge Router Provides:

• Rich Features, Flexible Service Capability

• Variety of Hardware to Meet Different Capacity Requirements

• Fully Distributed Architecture for High Performance and System Scalability

• Uniform, Open and Modularized Software Architecture for Simple Operation and Efficient Failure Protection

75BRKARC-2003


• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card.

• Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.

http://www.ciscolive.com/us

http://ciscolive.com/Online

http://ciscolive.com/Online

http://www.ciscolive.com/Online


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKARC-2003 77

Thank you

Date post:	21-Jan-2018
Category:	Internet
Upload:	bruno-teixeira
View:	1,476 times
Download:	11 times

Cisco Live! :: Cisco ASR 9000 Architecture :: BRKARC-2003 | Las Vegas 2017

Internet