+ All Categories
Home > Technology > Особенности архитектуры и траблшутинга...

Особенности архитектуры и траблшутинга...

Date post: 19-Jul-2015
Category:
Upload: cisco-russia
View: 606 times
Download: 4 times
Share this document with a friend
Popular Tags:
65
Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000 Дмитрий Леонтьев Инженер российского центра технической поддержки Cisco TAC 28.01.2015
Transcript
Page 1: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Дмитрий Леонтьев

Инженер российского центра технической поддержки Cisco TAC

28.01.2015

Page 2: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

2 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco Support Community – Expert Series Webcast

Сегодня на семинаре эксперт Cisco Дмитрий Леонтьев расскажет об особенностях архитектуры и траблшутинга

маршрутизаторов Cisco ASR1000 при его работе.

Дмитрий Леонтьев

Инженер российского

центра технической

поддержки Cisco TAC

Page 3: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

3 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Технические Эксперты

Тема: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Дата проведения вебинара: 28 января 2015 года

Олег Типисов

Инженер центра

технической поддержки

Cisco TAC в России

Сергей Василенко

Инженер центра

технической поддержки

Cisco TAC в России

Page 4: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

4 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Спасибо, что посетили наш вебинар сегодня

Сегодняшняя презентация включает опросы аудитории

Пожалуйста, участвуйте!

Page 5: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

5 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Спасибо, что присоединились к нам сегодня Скачать презентацию Вы можете по ссылке: https://supportforums.cisco.com/ru/document/12407596

Page 6: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Присылайте Ваши вопросы!

Используйте панель Q&A, чтобы задать вопрос. Наши эксперты ответят на них.

Page 7: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

7 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Вопрос №1

Какой у вас опыт в использовании ASR1000?

a) Только начинаю осваивать ASR1k и IOS-XE

b) Хорошие знания IOS-XE, но знакомство с ASR1k только начинается

c) Хорошие знания IOS-XE и уверенно пользуюсь системой

d) Давно использую ASR1K и являюсь специалистом

Page 8: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

8 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco Support Community – Expert Series Webcast

Дмитрий Леонтьев

Инженер российского центра технической поддержки Cisco

Январь, 2015

Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Page 9: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

9 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Архитектура маршрутизаторов ASR1000

Архитектура ASR1001, ASR1001-X и ASR1002-X

Прохождение пакета через маршрутизатор

Архитектура операционной системы IOS-XE

Примеры распространенных проблем на ASR1000 и способы их устранения

Программа вебинара

Page 10: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

10 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco 7200, 10+ years in networks

Page 11: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

11 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Goodbye, Cisco 7200

http://www.cisco.com/c/en/us/products/collateral/routers/7200-series-

routers/end_of_life_c51-681414.html

Milestone Definition Date

End-of-Life Announcement Date The date the document that announces the end-of-

sale and end-of-life of a product is distributed to the

general public.

September 30, 2011

End-of-Sale Date The last date to order the product through Cisco

point-of-sale mechanisms. The product is no longer

for sale after this date.

September 29, 2012

End-of-Sale Product

Part Number

Product Description Replacement Product Part

Number

Replacement Product

Description

Additional Information

CISCO7201 Cisco 7201 Chassis, 1GB

Memory, Dual P/S, 256MB

Flash

ASR1001 Cisco ASR1001 System,

Crypto, 4 built-in GE, Dual P/S

-

CISCO7201 Cisco 7201 Chassis, 1GB

Memory, Dual P/S, 256MB

Flash

ASR1002 Cisco ASR1002 Chassis, 4

built-in GE, Dual P/S, 4GB

DRAM

-

CISCO7201-5PK 5-Pack Bundle of CISCO7201

MRBU Router

See Product Migration Options

section for details.

ASR 1000 -

CISCO7204VXR Cisco 7204VXR, 4-slot

chassis, 1 AC Supply w/IP

Software

ASR1001 Cisco ASR1001 System,

Crypto, 4 built-in GE, Dual P/S

-

CISCO7204VXR Cisco 7204VXR, 4-slot

chassis, 1 AC Supply w/IP

Software

ASR1002 Cisco ASR1002 Chassis, 4

built-in GE, Dual P/S, 4 GB

DRAM

-

Page 12: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

12 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco 7200 and ASR 1000 Comparison

Характеристика Cisco 7200 ASR 1000

Throughput Up to 1.8 Gbps,

Up to 2Mpps

2.5-100 Gbps,

до 32Mpps

IP CEF Software Hardware

Operation system IOS IOS-XE

Page 13: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

13 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco 7200 Architecture

Power

Supply CPU

ROM RAM

Flash

Bus

Interface

Interface Interface

System Bus

NVRAM

Network

Controller

Network

Controller

Interface

Network

Controller

System Bus

Page 14: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

14 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Running all code in processes can be slow.

• Have to wait for the scheduler to allow the process to run.

• In IOS a process can not interrupt another process while

it’s running. The process has to wait until the other process

completes

• Processes have to be programmed to be well behaved

• CPU’s provide a mechanism to do processing on critical

things nearly immediately - Interrupts

Most interrupts are hardware initiated

Most CPU’s provide at least 1 software interrupt

Interrupting the CPU

Page 15: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

15 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ASR 1000 – Product Positioning

3845

7200 < 3G

>300G

List Price Price includes Chassis, engine

18G

7304-NSE

ASR1004 w/

ASR1000-ESP20

ASR1002 w/

ASR1000-ESP5

ASR1006 w/ dual

ASR1000-ESP10

ASR1000-RP1

5G

10G

20G 7600,

GSR,CRS

ASR 1000 Series

5-200Gbps (Depends on ESP/SIP not Chassis Type)

ASR1002 w/

ASR1000-ESP10

Page 16: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

16 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ASR 1000 Series Building Blocks

Route

Processor (standby)

RP

Interconn.

Embedded Services Processor

(active)

FECP

Interconn.

QFP subsys-

tem Crypto assist

Embedded Services Processor (standby)

FECP

Interconn.

QFP subsystem Crypto

assist

SPA SPA

IOCP SPA Agg.

Interconn.

SPA SPA

IOCP SPA Agg.

Interconn.

SPA SPA

IOCP SPA Agg.

Interconn.

Passive Midplane

Route Processor

(active)

RP

Interconn.

Hypertransport, 10Gbps

ESI, (Enhanced Serdes) 11.5Gbps SIPs ESI, (Enhanced Serdes) 11.5Gbps

• RP (Route Processor)

Handles control plane traffic

Manages system

• ESP

Handles forwarding plane traffic

• SPA Interface Processor

Shared Port Adapters provide interface connectivity

• Centralized Forwarding Architecture

All traffic flows through the active ESP, standby is synchronized with all flow state with a dedicated ESI link

• Distributed Control Architecture

All major system components have a powerful control processor dedicated for control and management planes

Page 17: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

17 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ASR1000 Control Plane Links

• Ethernet out-of-band Channel (EOBC)

Run between ALL components

Indication if cards are installed and ready

Watchdog timers

State information exchange for L2 or L3 Protocols

• Inter-Integrated Circuit (I2C)

Monitor health of hardware components

Control resets

Communicate acive/standby, Real time presence and ready indicators

Control the other RP (reset, power-down,interrupt, report Power-supply status, signal ESP active/standby)

EEPROM access

• SPA control links

Run between IOCP and SPAs

Detect SPA OIR

Reset SPAs (via I2C)

Power-control SPAs (via I2C)

Read EEPROMs

SPA SPA

IOCP SPA

Agg.

Interconn.

SPA SPA

IOCP SPA

Agg.

Interconn.

SPA SPA

IOCP SPA

Agg.

Interconn.

Route Processor (Standby)

Route Processor

(active)

Forwarding Processor (Standby)

FECP

Interconn.

QFP subsys-

tem Crypto assist

Forwarding Processor

(active)

FECP

Interconn.

QFP subsys-

tem Crypto assist

Midplane

RP RP

GE, 1Gbps

I2C

SPA Control

SPA Bus

SPA SPA

IOCP SPA

Agg.

Interconn.

SPA SPA

IOCP SPA

Agg.

Interconn.

SPA SPA

IOCP SPA

Agg.

Interconn.

Page 18: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

18 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

FP = Forwarding Processor now known as the Embedded Services

Processor (ESP)

CPP = Cisco Packet Processor now known as the QuantumFlow

Processor (QFP)

PPEs = Packet Processor Elements

CC = CarrierCard now known as the SPA Interface Processor (SIP)

RP = Route Processor

FRU = Field Replaceable Unit

IOSd = IOS daemon, IOS process running on RP

ASR1000 - Naming

Page 19: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

19 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco ASR1000 Series

ASR1001 ASR 1002 - F ASR 1002 ASR 1004 ASR 1006 ASR 1013

Chassis

Scalable to 5 Gbps

Four built-in GE ports

Software redundancy

Scalable to 2.5 Gbps

Four built-in GE ports

Software redundancy

Scalable to 10 Gbps

Four built-in GE ports

Software

redundancy

Scalable to 40 Gbps

Software

redundancy

Scalable to 100 Gbps

Hardware

redundancy

Scalable to 200Gbps

Hardware

redundancy

Embedded

Services

Processors

Integrated Software

Upgradeable

ASR1001-ESP2.5/5

(single)

Integrated ESP

(2.5-Gbps)

(single)

ASR1000-ESP5

(single)

ASR1000-ESP10

(single)

ASR1000-ESP10

(single)

ASR1000-ESP20

(single) ASR1000-ESP40

(single)

ASR1000-ESP10

(redundant-optional)

ASR1000-ESP20

(redundant-optional) ASR1000-ESP40

(redundant-optional)

ASR1000-ESP100

(redundant-optional)

ASR1000-ESP40

(redundant-optional) ASR1000-ESP100

(redundant-optional)

ASR1000-ESP200

(redundant-optional)

Route Processor

Integrated

ASR1001-RP

(single)

Integrated

ASR1000-RP1

(single)

Integrated

ASR1000-RP1

(single)

ASR1000-RP1

(single)

ASR1000-RP2

(single)

ASR1000-RP1

(redundant-optional)

ASR1000-RP2

(redundant-optional)

ASR1000-RP2

(redundant-optional)

SPA Interface

Processor Integrated SIP10 Integrated SIP10 Integrated SIP10

ASR1000-SIP10

ASR1000-SIP40

ASR1000-SIP10

ASR1000-SIP40

ASR1000-SIP10

ASR1000-SIP40

SPA Slots 1 (single-height) 1 (single-height) 3 8 12 24

Page 20: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

20 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco ASR1000 Series

ASR1001 ASR 1001 - X ASR 1002 ASR 1002 - X

Chassis

Scalable to 5 Gbps

Four built-in GE ports

Software redundancy

Scalable to 20 Gbps

Six built-in GE ports +

Two built-in 10GE ports

Software redundancy

Scalable to 10 Gbps

Four built-in GE ports

Software redundancy

Scalable to 40 Gbps+

Software redundancy

Embedded

Services

Processors

Integrated Software

Upgradeable

ASR1001-ESP2.5/5

(single)

Integrated Software

Upgradeable

ASR1001-X (2.5/5/10/20)

(single)

ASR1000-ESP5

(single)

ASR1000-ESP10

(single)

Integrated Software

Upgradeable

ASR1002-X (5/10/20/36)

(single)

Route Processor

Integrated

ASR1001

(single)

Integrated

ASR1001-X

(single)

Integrated

ASR1000-RP1

(single)

Integrated

ASR1002-X

(single)

SPA Interface

Processor Integrated SIP10 Integrated ASR1001-X Integrated SIP10

Integrated ASR1002-X

SPA Slots 1 (single-height) 1 (single-height) 3 3

Page 21: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

21 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Chassis Options: ASR1006

RP1 in slots “r0”&“r1”

ESP10

SIP10

SPAs

Rack Mounts and Cable Mgt not shown

Page 22: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

22 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Numbering Convention

USB 0 USB 1

PWR 1

PWR 0

SIP 2

F0

SIP 0

F1

SIP 1

R1

R0

Page 23: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

23 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

------------------ show platform ------------------

Chassis type: ASR1006

Slot Type State Insert time (ago)

--------- ------------------- --------------------- -----------------

0 ASR1000-SIP10 ok 1d01h

0/0 SPA-2X1GE-V2 ok 1d01h

1 ASR1000-SIP10 ok 1d01h

1/0 SPA-8X1GE-V2 ok 1d01h

R0 ASR1000-RP2 ok, active 1d01h

R1 ASR1000-RP2 ok, standby 1d01h

F0 ASR1000-ESP20 ok, active 1d01h

F1 ASR1000-ESP20 ok, standby 1d01h

P0 ASR1013/06-PWR-AC ok 1d01h

P1 ASR1013/06-PWR-AC ok 1d01h

Slot CPLD Version Firmware Version

--------- ------------------- ---------------------------------------

0 14011701 15.4(2r)S

1 14011701 15.4(2r)S

R0 13092401 15.2(1r)S

R1 13092401 15.2(1r)S

F0 12071700 15.3(1r)S

F1 12071700 15.3(1r)S

Hardware configuration checking

Page 24: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

24 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

RP Functions Summary • CPU

Runs IOS

Control plane processing

Chassis management (via EOBC GE Switch, Inter-Integrate Circuit Mux…)

Activation / de-activation of other building blocks

Alarm handling

Logging

HA switchover

Image management

Processes legacy protocols (punt path)

• Console / Aux / Management Ethernet

Provide management access interfaces

Aux does not provide a normal CLI

• Interconnect

Provides interconnection for data path (via ESI)

• GE Switch

Provides interconnection to other building blocks (IPC over EOBC)

• I2CMux

Control plane interconnection to other building blocks (e.g. used for resets)

Page 25: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

25 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

RP1 and RP2 Comparison

ASR1000 RP1 ASR1000 RP2

CPU 1.5GHz Dual-Core 2.67GHz

Memory 2GB default (2x1GB)

4GB maximum (2x2GB)

RP1 with 4GB built in

ASR1002 and

ASR1002-F

8GB default (4x2GB)

16GB maximum

(4x4GB)

Built-in eUSB bootflash 1GB (8GB on ASR-1002

and ASR1002-F)

2GB

NVRAM 32MB 32MB

Hard disk drive size 40GB 80GB

Chassis Support ASR 1002 (built-in),

ASR 1004 and ASR

1006

ASR 1004 and ASR

1006 and ASR1013

Cisco IOS XE Operating

System

32 bit 64 bit

Page 26: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

26 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ESP Functions

• Integrates QuantumFlow Processor Architecture

Data plane forwarding / Data plane services / QFP Client

MAC classification

L2/L3 forwarding

QoS

ACL

VPN

Netflow etc. etc. etc. etc. etc. etc.

• FECP

Manages ESP (Crypto, QFP, …)

Communicates with RP using IPC over EOBC

Statistics reporting to RP

OBFL

• Crypto

Processes security functions for data plane packets

• Data Interconnect to SIPs, RPs and other ESP if present

Enhanced SerDes Interconnect (ESI)

Supports 2 Data priorities + Control

• SPI Mux – System Packet Interface’s give 10Gbps BW

provides connectivity between QFP, Crypto and the Interconnect

Same standard interface used to connect to SPAs

Page 27: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

27 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Embedded Services Processors (ESP)

• Centralized, programmable forwarding engine (i.e. QFP subsystem (PPE) and crypto engine) providing full-packet processing

• Packet buffering and queuing/scheduling (BQS)

For output traffic to carrier cards/SPA’s

For special features such as input shaping, reassembly, replication, punt to RP, etc.

• Interconnect providing data path links (ESI) to/from other cards over midplane

Transports traffic into and out of the Cisco QuantumFlow Processor (QFP)

Input scheduler for allocating QFP BW among ESI’s

• FECP CPU managing QFP, crypto device, midplane links, etc

ESP10

Page 28: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

28 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco QuantumFlow Processor (QFP)

Multi-Core (40) Packet Processor

Traffic Manager

+ + QuantumFlow

Processor

Software

Page 29: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

29 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ESP Generations

ESP-2.5G ESP-5G ESP-10G ESP-20G ESP-40G ESP-100G ESP-200G

System Bandwidth

2.5Gbps 5Gbps 10Gbps 20Gbps 40Gbps 100Gbps 200Gbps

# of Processors (PPEs)

10 20 40 40 40 124 248

Clock Rate 900 Mhz 900 Mhz 900 Mhz 1.2 GHz 1.2 GHz 1.5 GHz 1.5 GHz

Crypto Engine BW (1400 bytes)

1Gbps 1.8Gbps 4.4Gbps 8.5Gbps 11Gbps 29Gbps 78Gbps

QFP Resource Memory

256MB 256MB 512MB 1GB 1GB 4GB 8GB

Packet Buffer 64MB 64MB 128MB 256MB 256MB 1GB 2GB

Control CPU 800 MHz 800 MHz 800 MHz 1.2 GHz Dual core 1.86 GHz

Dual core 1.73 GHz

Dual core 1.73 GHz

Control Memory

1GB 1GB 2GB 4GB 8GB 16GB 32GB

TCAM 10Mb 10Mb 10Mb 40Mb 40Mb 80Mb 2x80Mb

Chassis Support

ASR 1002-F (Integrated)

ASR 1002 ASR 1002, 1004, 1006

ASR 1004, 1006

ASR 1004, 1006, 1013

ASR1006, 1013

ASR1013

Page 30: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

30 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

SIP10 and SIP40 Comparison

ASR1000-SIP10 ASR1000-SIP40

Bandwidth 10G 40G

Ingress Buffering 128MB 128MB

Egress Buffering 8MB 8MB

Bandwidth per ESI Link 11Gbps 23Gbps

ESI Links used 1 1 or 2

Page 31: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

31 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

SIP10 Functions

• SPA Aggregation

Provides connectivity for data plane to SPAs

Ingress classification and buffering

Egress buffering

Reporting of egress queue status to ESP

• IOCP

Runs separate SPA driver for each SPA

Initialization, configuration, statistics, link handling events

Manages Carrier Card components

Handles OIR

Communicates with RP via IPC/EOBC

OBFL

• Interconnect

Provides Interconnection for the data path

• Network timing circuitry

Responsible for timing

Generate local clock

Page 32: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

32 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

g

Interconn.

Ingress classifier

Ingress Scheduler

Egress Buffer Status

ESI, 10/40Gbps

SPA-SPI, 11.2Gbps

Hypertransport, 10Gbps

Other

4 SPAs

Ingress Buffers (per port)

Egress Buffers (per port)

ESPs

SPA Agg.

SPA

aggregation

ASIC

Data

32

1. SPA receives packet data from its network

interfaces and transfers the packet to the SIP

2. SPA Aggregation ASIC classifies the

packet into H/L priority

3. SIP writes packet data to external 128B memory

(at 40Gbps from 4 full-rate SPAs)

4. Ingress buffer memory is carved into 64 queues.

The queues are arranged by SPA-SPI channel

and optionally H/L. Channels on “channelized”

SPAs share the same queue.

5. SPA ASIC selects among ingress queues for

next pkt to send to ESP over ESI. It prepares

the packet for internal transmission

6. The interconnect transmits packet data of

selected packet over ESI to active ESP at up to

40 Gbps

7. Active ESP can backpressure SIP via ESI ctl

message to slow pkt transfer over ESI if

overloaded (provides separate backpressure

for Hi vs. Low priority pkt data)

Data Packet Flow: From SPA Through SIP

Page 33: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

33 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Data Packet Flow: Through ESP40 1. Packet arrives on QFP

2. Packet assigned to a PPE thread.

3. The PPE thread processes the packet in a feature chain similar to 12.2S IOS (very basic view of a v4 use case):

Input Features applied

NetFlow, MQC/NBAR Classify, FW, RPF, Mark/Police, NAT, WCCP etc.

Forwarding Decision is made

Ipv4 FIB, Load Balance, MPLS, MPLSoGRE, Multicast etc.

Output Features applied

NetFlow, FW, NAT, Crypto, MQC/NBAR Classify, Police/Mark etc.

Finished

4. Packet released from on-chip memory to Traffic Manager (Queued)

5. The Traffic Manager schedules which traffic to send to which SIP interface (or RP or Crypto Chip) based on priority and what is configured in MQC

6. SIP can independently backpressure ESP via ESI control message to pace the packet transfer if overloaded

Interconnect

Pkt Buffer DRAM

(128MB)

Part Len/ BW SRAM

Resource DRAM

(512MB)

SIP-10

TCAM4 (10Mbit)

Processor pool

PPE0 PPE0 PPE0 PPE1

PPE0 PPE0 PPE0 PPE6

PPE0 PPE0 PPE0 PPE2

PPE0 PPE0 PPE0 PPE5

PPE0 PPE0 PPE0 PPE3

… PPE0 PPE0 PPE0 PPE40

PPE0 PPE0 PPE0 PPE4

Buffer, queue, schedule (BQS)

Quantum Flow Processor

Buffer, queue, schedule (BQS)

Buffer, queue, schedule (BQS)

Dispatcher/Pkt Buffer

Data

ESI, 10/40Gbps

SPA-SPI, 11.2Gbps

Hypertransport, 10Gbps

Other

ASR System BW (Depends on ESP)

Page 34: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

34 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

g

Interconn.

Ingress classifier

Ingress Scheduler

Egress Buffer Status

ESI, 10/40Gbps

SPA-SPI, 11.2Gbps

Hypertransport, 10Gbps

Other 4 SPAs

Ingress Buffers (per port)

Egress Buffers (per port)

ESPs

SPA Agg.

SPA

Aggregation

ASIC

Data Packet Flow: Through SIP to SPA

1. Interconnect receives packet data over ESI from the active ESP at up to 40 Gbps

2. SPA Aggregation ASIC receives the packet and writes it to external egress buffer memory

3. Egress buffer memory is carved into 64 queues. The queues are arranged by egress SPA-SPI channel and optionally H/L. Channels on “channelized” SPAs share the same queue.

4. SPA Aggregation ASIC selects and transfers packet data from eligible queues to SPA-SPI channel (Hi queue are selected before Low)

5. SPA can backpressure transfer of packet data burst independently for each SPA-SPI channel using SPI FIFO status

6. SPA transmits packet data on network interface

Data

Page 35: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

35 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ASR 1002 & 1002-X, 1001, 1001-X Differences

ASR1002

• Each of the 3 main building blocks has its own control CPU

Route Processor (RP1)

Embedded services Processor (ESP5/ESP10)

SPA interface Processor (SIP10)

• Comes with 4GB DRAM (default and maximum)

• 32bit Architecture

ASR1002-X, ASR1001, ASR1001-X • All 3 main building blocks are controlled by one CPU

RP, SIP, ESP

Default system bandwidth of 5 Gbps upgradable to 10, 20, 36 Gbps via software licenses

• Comes with 4GB DRAM (default); upgradeable to 8GB and 16GB DRAM. 16GB DRAM is maximum. (ASR1002-X, ASR1001)

• Comes with 8GB DRAM (default); upgradeable to 16GB DRAM. 16GB DRAM is maximum. (ASR1001-Х)

• 64bit Architecture

Page 36: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

36 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ASR1000 Software Architecture – IOS XE

• IOS XE = IOS + IOS XE Middleware + Platform Software

• Operational Consistency - same look and feel as IOS Router

• IOS runs as its own Linux process for control plane (Routing, SNMP, CLI etc). Capable of 64bit operation

• Linux kernel with multiple processes running in protected memory for

Fault containment

Re-startability

ISSU of individual SW packages

• ASR 1000 HA Innovations

Zero-packet-loss RP Failover

<50ms ESP Failover

“Software Redundancy” SPA Interface Processor Embedded Services

Processor

Route Processor

Control Messaging

Kernel Kernel

Kernel

QFP

Client/Driver

Chassis

Manager

Interface

Manager Forwarding

Manager

SPA

Driver

SPA

Driver

SPA

Driver

SPA

Driver

IOS

(Standby)

Forwarding

Manager

Chassis

Manager

IOS

(Active)

IOS XE Platform Adaptation Layer (PAL)

Interface

Manager

Chassis

Manager

Page 37: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

37 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco.com->Support->Download

Page 38: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

38 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Support Model Cisco IOS XE Software Release ASR 1000

Standard-Support

2.1, 2.2, 2.3, 2.5, 2.6, 3.2S, 3.3S, 3.5S, 3.6S, 3.8S,

3.9S

Extended-Support 2.4, 3.1S, 3.4S, 3.7S

Cisco IOS XE Software Release 3.7S End-of-Life Timeline

Cisco IOS XE Software Release Support

Page 39: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

39 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Support Model Cisco IOS XE Software Release ASR 1000

Standard-Support

3.11S, 3.12S, 3.14S, 3.15S, etc.

Extended-Support 3.10S, 3.13S, 3.16S, etc.

Cisco IOS XE Software Release 3.10S is the first Extended-Support release

under the new support timeline. Every subsequent third release (for example,

Cisco IOS XE Software Releases 3.13S, 3.16S, etc.) will also be Extended-

Support releases.

Cisco IOS XE Software Release Support

Page 40: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

40 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco IOS XE Software Standard-Support Schedule

Cisco IOS XE Software Standard-Support releases will be supported for 18

months with three rebuilds. The timing of the rebuilds will be as follows: 3 months,

3 months, and 6 months. Following the third rebuild will be a 6-month phase of

PSIRT. During the PSIRT phase, if it is deemed that a fourth rebuild is required,

then a scheduled rebuild will occur at the end of the Cisco Product Security

Incident Response Team (PSIRT) period (6 months from the third rebuild).

Page 41: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

41 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco IOS XE Software Extended-Support Schedule

Cisco IOS XE Software Extended-Support releases will be supported for 48

months with eight rebuilds. The timing of the rebuilds will be as follows: 3

months, 3 months, 4 months, 4 months, 4 months, 6 months, 6 months, and 6

months. Following the eighth rebuild will be a 12-month phase of PSIRT. During

the PSIRT phase, if it is deemed that additional rebuilds are required, then a

scheduled rebuilds will occur 6 months into the PSIRT phase, or another at the

end of the PSIRT phase. If it is deemed that these additional rebuilds are not

required, then there will not be additional rebuilds.

Page 42: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

42 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Recommended ROMmon Release for Each FRU

------------------ show platform ------------------

......

Slot CPLD Version Firmware Version

--------- ------------------- ---------------------------------------

0 14011701 15.4(2r)S

1 14011701 15.4(2r)S

R0 13092401 15.2(1r)S

R1 13092401 15.2(1r)S

F0 12071700 15.3(1r)S

F1 12071700 15.3(1r)S

http://www.cisco.com/c/en/us/td/docs/routers/asr1000/release/notes/asr1k_rn_rel_notes/asr1k_rn_sys_req.html#pgfId-3201398

Page 43: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

43 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Вопрос №2

a) Как BRAS/ISG

b) Настроен NAT/PAT

c) Настроен IPSec/DMVPN

d) Как маршрутизатор для подключения к провайдеру (BGP)

e) Выжимаю из него все что умеет

Как Вы используете ASR1000?

Page 44: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

44 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ASR 1000 Memory Allocation

RP and Physical Memory Memory Allocated to IOSd

(w/o IOSd redundancy enabled)

Memory Allocated to Kernel

and other processes

RP1 (4GB) 1.7GB 2.3GB

RP2 (8GB) 4.2GB 3.8GB

RP2 (16GB) 10GB 6GB

ASR 1001 – 4GB 1.4GB 2.6GB

ASR 1001 – 8GB 4GB 4GB

ASR 1001 – 16GB 7GB 9GB

• Memory allocation is fixed by design, not configurable.

• If turn on IOSd redundancy, memory allocated to each IOSd is further reduced by more than half comparing to memory allocation without IOSd redundancy enabled

• ASR 1001 memory is shared among RP, ESP, SIP

Page 45: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

45 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

ASR 1000 System Resources To Monitor

Page 46: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

46 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

show process memory output (IOS)

ASR1002

Processor Pool Total: 1695268496 Used: 362330804 Free: 1332937692

lsmpi_io Pool Total: 6295088 Used: 6294116 Free: 972

ASR1002-X

Processor Pool Total: 3905088960 Used: 2876800024 Free: 1028288936

lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

Page 47: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

47 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

show version output (IOS)

ASR1002

cisco ASR1002 (2RU) processor with 1655643K/6147K bytes of memory.

6 Gigabit Ethernet interfaces

32768K bytes of non-volatile configuration memory.

4194304K bytes of physical memory.

7798783K bytes of eUSB flash at bootflash:.

ASR1002-X

cisco ASR1002-X (2RU-X) processor with 3813739K/6147K bytes of memory.

6 Gigabit Ethernet interfaces

32768K bytes of non-volatile configuration memory.

8388608K bytes of physical memory.

6684671K bytes of eUSB flash at bootflash:.

Page 48: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

48 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

show platform software status control-processor brief

ASR1002

Load Average

Slot Status 1-Min 5-Min 15-Min

RP0 Healthy 0.22 0.11 0.07

ESP0 Healthy 0.01 0.00 0.00

SIP0 Healthy 0.00 0.00 0.00

Memory (kB)

Slot Status Total Used (Pct) Free (Pct) Committed (Pct)

RP0 Healthy 3874476 1834856 (47%) 2039620 (53%) 2478476 (64%)

ESP0 Healthy 2009400 804956 (40%) 1204444 (60%) 586828 (29%)

SIP0 Healthy 471804 350112 (74%) 121692 (26%) 300896 (64%)

CPU Utilization

Slot CPU User System Nice Idle IRQ SIRQ IOwait

RP0 0 8.28 2.89 0.00 88.62 0.09 0.09 0.00

ESP0 0 2.40 18.70 0.00 78.80 0.00 0.10 0.00

SIP0 0 3.60 1.40 0.00 94.90 0.00 0.10 0.00

Page 49: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

49 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

show platform software status control-processor brief

ASR1002-X

Load Average

Slot Status 1-Min 5-Min 15-Min

RP0 Healthy 0.08 0.07 0.07

Memory (kB)

Slot Status Total Used (Pct) Free (Pct) Committed (Pct)

RP0 Critical 8092408 8011928 (99%) 80480 ( 1%) 8284496 (102%)

CPU Utilization

Slot CPU User System Nice Idle IRQ SIRQ IOwait

RP0 0 8.00 2.40 0.00 89.58 0.00 0.00 0.00

1 0.40 0.60 0.00 98.99 0.00 0.00 0.00

2 0.10 0.20 0.00 99.70 0.00 0.00 0.00

3 0.00 0.00 0.00 100.00 0.00 0.00 0.00

Page 50: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

50 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

show bgp vpnv4 unicast all summary

BGP router identifier 10.10.10.10, local AS number 00000

BGP table version is 122579269, main routing table version 122579269

511585 network entries using 130965760 bytes of memory

1007045 path entries using 112789040 bytes of memory

544817/77417 BGP path/bestpath attribute entries using 135114616 bytes of memory

5 BGP rrinfo entries using 200 bytes of memory

142975 BGP AS-PATH entries using 6954234 bytes of memory

15542 BGP community entries using 1359838 bytes of memory

82 BGP extended community entries using 2660 bytes of memory

578 BGP route-map cache entries using 36992 bytes of memory

0 BGP filter-list cache entries using 0 bytes of memory

BGP using 387223340 total bytes of memory

BGP activity 5254146/4742553 prefixes, 81675934/80668874 paths, scan interval 60 secs

Page 51: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

51 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Cisco ASR 1002-X Integrated Route Processor Product Specifications

Item Details

Chassis support Cisco ASR 1002-X chassis

Software compatibility Cisco IOS XE Software Release 3.7.0S and later versions

Performance With 4-GB memory:

● Up to 500,000 IPv4 or 500,000 IPv6 routes

With 8-GB or more memory:

● Up to 1,000,000 IPv4 or 1,000,000 IPv6 routes – 8GB

Memory

● Up to 3,500,000 IPv4 or 3,000,000 IPv6 routes – 16GB

Memory

● BGP RR Scalability up to 7,000,000 IPv4 or 6,000,000

IPv6 routes – 8GB Memory

● BGP RR Scalability up to 13,000,000 IPv4 or 11,000,000

IPv6 routes – 16GB Memory

http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-

services-routers/data_sheet_c78-441072.html

Page 52: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

52 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Software redundancy

#show running-config

...

redundancy

mode sso

#show version

...

cisco ASR1004 (RP2) processor with 1575783K/6147K bytes of memory.

8 FastEthernet interfaces

8 Gigabit Ethernet interfaces

4 Packet over SONET interfaces

32768K bytes of non-volatile configuration memory.

8388608K bytes of physical memory.

1873919K bytes of eUSB flash at bootflash:.

78085207K bytes of SATA hard disk at harddisk:.

Page 53: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

53 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Alarm LED Is Illuminated

If the CRIT, MAJ, or MIN alarm LED is illuminated, determine the cause of the alarm by doing one of the following:

• Review the alarm message. The logging alarm command must be enabled for the system to send alarm messages to the console. The following is an example of an alarm message that was generated when a SPA was removed without a graceful deactivation of the SPA:

*Aug 22 13:27:33.774: %ASR1000_OIR-6-REMSPA: SPA removed from subslot 1/1, interfaces

disabled

*Aug 22 13:27:33.775: %SPA_OIR-6-OFFLINECARD: SPA (SPA-4XT-SERIAL) offline in subslot 1/1

• Enter the show facility-alarm status command. The following example shows a critical alarm that is generated when a SPA is removed fr om the system:

Router# show facility-alarm status

System Totals Critical: 1 Major: 0 Minor: 0

Source Severity Description [Index]

------ -------- -------------------

subslot 1/1 CRITICAL Active Card Removed OIR Alarm [0]

A critical alarm "Active Card Removed OIR Alarm" is generated even if a SPA is removed after performing graceful deactivation.

Page 54: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

54 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Critical Error LED indicator

#show facility-alarm status

System Totals Critical: 1 Major: 0 Minor: 0

Source Severity Description [Index]

------ -------- -------------------

GigabitEthernet0/0/0 CRITICAL Physical Port Link Down [1]

xcvr container 0/0/1 INFO Transceiver Missing [0]

xcvr container 0/0/2 INFO Transceiver Missing [0]

show run show interfaces

interface GigabitEthernet0/0/0

ip address 10.10.10.1 255.255.255.0

negotiation auto

!

interface GigabitEthernet0/0/1

no ip address

shutdown

negotiation auto

!

GigabitEthernet0/0/0 is down, line protocol is down

GigabitEthernet0/0/1 is administratively down, line

protocol is down

Page 55: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

55 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Critical Error LED indicator

Solution

Now that you know what is causing the alarm in this case you basically have two choices. You can place the interface that is causing the status into admin down until you turn up the interface and place it into testing or production. (ie in config-interface perform shutdown) This is preffered. You could also just ignore the error, although it not suggested unless you're going to be turning up the interface in the next couple minutes. If you have another critical issue you won't know the difference, via the LED indicators in any case.

Finally, clearing the alarms with clear facility-alarm most likely won't work unless the state of the interface(s) has changed either to admin down or up/up because the alarm will just be thrown again in a couple of seconds. When the interface(s) are in a preferred state this alarm (LED indicator) should clear automatically.

Page 56: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

56 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Automatic Router Shutdown When the router detects a condition that could result in physical damage to system components, the router can shut down without operator intervention. When the router shuts down automatically, the system controller disables DC power to all internal components. All DC power remains disabled until you toggle the power switch.

The default for automatic router shutdown is off. To allow automatic router shutdown, the facility-alarm critical exceed-action shutdown command must be enabled. If the facility-alarm critical exceed-action shutdown command is enabled, the router performs an automatic shutdown under the following conditions:

Internal Temperature of Router or Power Supply Exceeds Temperature Threshold

Voltage of AC or DC Power Supplies Is Out of Tolerance

Power Supply Is Removed

Two power supplies must be installed in the chassis at all times to ensure sufficient cooling. The system fans are inside the power supply units and must spin for cooling. Because all the system fans can be powered by one power supply, the second power supply unit does not have to be powered on, but it must be installed.

If a power supply is removed, the system can run with only one power supply for a maximum of five minutes. The router waits five minutes before shutting down. This five-minute window allows time to replace a failed power supply.

Two power supplies are not required but recommeneded for Cisco ASR 1001Router. An automatic 5 minute shutdown will not occur if power supply is removed for Cisco ASR 1001 Router.

Page 57: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

57 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Embedded Packet Capture

Embedded Packet Capture (EPC) is an onboard packet capture facility that allows network administrators to capture packets flowing to, through, and from the device and to analyze them locally or save and export them for offline analysis by using a tool such as Wireshark. This feature simplifies network operations by allowing devices to become active participants in the management and operation of the network. This feature facilitates troubleshooting by gathering information about the packet format.

#monitor capture cisco interface GigabitEthernet0/0/0 both

#monitor capture cisco buffer circular size 100

#monitor capture cisco match any

#monitor capture cisco start

#monitor capture cisco stop

#monitor capture cisco export tftp://10.10.10.10/cisco.pcap

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/epc/configuration/xe-3s/asr1000/epc-xe-3s-asr1000-book/nm-packet-capture-xe.html

Page 58: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

58 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Embedded Event Manager

Embedded Event Manager (EEM) is a distributed and customized approach to event detection and recovery offered directly in a Cisco IOS XE device. EEM offers the ability to monitor events and take informational, corrective, or any desired EEM action when the monitored events occur or when a threshold is reached. An EEM policy is an entity that defines an event and the actions to be taken when that event occurs.

event manager applet if_stats

event timer cron cron-entry "0,10,20,30 12 28 01 *"

action 1.02 cli command "enable"

action 1.03 cli command "show clock | append bootflash:if_stats.log"

action 1.04 cli command "show process cpu sorted | append bootflash:if_stats.log"

action 1.05 cli command "show interfaces | append bootflash:if_stats.log"

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/eem/configuration/xe-3s/asr1000/eem-xe-3s-asr1000-book/eem-overview.html

Page 59: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

59 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Вопрос №3

С какими проблемами вы сталкивались при работе с ASR1000

a) Некорректная работа BRAS/ISG

b) Некорректная работа NAT/PAT

c) Некорректная работа IPSec/DMVPN

d) Нехватка памяти

e) Крэш маршрутизатора

f) Проблем в работе не замечено

Page 60: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Отправьте свой вопрос сейчас!

Используйте панель Q&A, чтобы задать вопрос. Эксперты ответят на Ваши вопросы.

Page 61: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Получить дополнительную информацию, а также задать вопросы эксперту в рамках данной темы Вы можете на странице, доступной по ссылке:

https://supportforums.cisco.com/community/russian/expert-corner

Вы можете получить видеозапись данного семинара и текст сессии Q&A в течении ближайших 5 дней по следующей ссылке

https://supportforums.cisco.com/community/russian/expert-corner/webcast

Page 62: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

62 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

Вебинар на русском языке

Тема: Outbound Option Dialer в UCCX/UCCE - конфигурация, поиск и устранение неисправностей

во вторник, 17 февраля, в 12.00 мск Присоединяйтесь к эксперту АМТ-ГРУП

Ирине Букреевой

Во время презентации инженер центра технической

поддержки АМТ-ГРУП расскажет о поиске и устранении

неисправностей систем исходящего обзвона в Cisco Unified

Contact Center Enterprise/Express. Вы сможете узнать об

архитектуре, особенностях конфигурации, а также будут

рассмотрены примеры анализа логов для Outbound Option

Dialer.

Page 63: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

63 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

https://supportforms.cisco.com/community/russian

http://www.facebook.com/CiscoSupportCommunity

http://twitter.com/#!/cisco_support

http://www.youtube.com/user/ciscosupportchannel

https://plus.google.com/110418616513822966153?prsrc=3#110418616513822966

153/posts

http://itunes.apple.com/us/app/cisco-technical-support/id398104252?mt=8

https://play.google.com/store/apps/details?id=com.cisco.swtg_android

http://www.linkedin.com/groups/CSC-Cisco-Support-Community-3210019

Newsletter Subscription: https://tools.cisco.com/gdrp/coiga/showsurvey.do?surveyCode=589&keyCode=146298_2&PH

YSICAL%20FULFILLMENT%20Y/N=NO&SUBSCRIPTION%20CENTER=YES

Page 64: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

64 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

• Испанском https://supportforums.cisco.com/community/spanish

• Португальском https://supportforums.cisco.com/community/portuguese

• Японском https://supportforums.cisco.com/community/csc-japan

Page 65: Особенности архитектуры и траблшутинга маршрутизаторов серии ASR1000

Спасибо за Ваше время

Пожалуйста, участвуйте в опросе


Recommended