Date post: | 20-Oct-2015 |
Category: |
Documents |
Upload: | helios-de-creisquer |
View: | 121 times |
Download: | 8 times |
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Best Practices to Deploy
High-Availability in Service Provider
Edge and Aggregation Architectures
BRKSPG-2402
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Abstract
As Service Providers are deploying value-added triple-play or quadruple play services to maintain or generate a higher average revenue per
user, overall Service Availability becomes increasingly important. High Availability techniques such as Fast Convergence or MPLS TE FRR
have focused on raising the availability of the network core in the past. Recently, these techniques are being increasingly deployed in
Ethernet Aggregation networks, for example by introducing MPLS TE FRR in the aggregation. Also, additional high-availability mechanism are
being developed to enhance the resilience of the IP Edge against failures. Examples of new developments include IP Fast-Reroute, BGP
Prefix Independent Convergence for both the Core and Edge, or even stateful application inter-chassis redundancy mechanisms to overcome
single-system outages. This Session aims to provide the audience with best current practices to increase service availability by deploying
Cisco High-Availability mechanisms in both the Aggregation and the IP Edge. Traditional HA techniques such as NSF/SSO, BFD, Fast
convergence or NSR are reviewed. The details of new technologies such as IP FRR, BGP PIC are discussed in depth. Furthermore,
advanced topics such as achieving HA for Layer 4-7 services or stateful inter-chassis redundancy solutions are introduced. The Session also
provides the best current practices of deploying the tools offered by the Cisco High-availability toolset, in particular the deployment of MPLS
TE FRR in the aggregation. Furthermore, possible stateful and stateless clustering approaches are introduced, which SPs may use to
increase the availability of their IP Edge architecture.
For Your Reference
3
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Glossary NHAT next hop address tracking EOBC Ethernet out of band management ACL Access Control List ESP Embedded Services Processor ACT Active EVC Ethernet Virtual Circuit APS Automatic Protection Switching EVDO Evolution Data Only ARP Address Resolution Protocol FECP Forwarding Engine Control Processor AS autonomous System FIB Forwarding Information Base ATM Asynchronous Transfer Mode FM Forwarding Manager BFD Bi Directional Forwarding Detection FR Frame Relay BNG Broadband Network Gateway FRR Fast Re Route
BW Bandwidth FSOL First Sign of Life CC Continuity Check FWLB Firewall Loadbalancing CC control connection GEC Gigabit Ether Channel CDR call detail record GLBP Global Load Balancing Protocol CE Customer Edge GR Graceful Restart CE Customer Edge GRE Generic Route Encapsulation CF checkpoint facility GW Gateway CFM Configuration and Fault Management HA High Availability CLI Command Line Interface HSRP Hot Standby Routing Protocol CM Chassis Manager HW Hardware CP Control Plane IETF Internet Engineering Task Force CPLD Complex Programmable Logic Device ? IF Interface CSC Carrier's Carrier IGP Internal Gateway Protocol DHCP Dynamic Host Configuration Protocol IOCP Input Output control Processor DP Data Plane IOS Internet Operating System DPM Defects per Million IP Internet Protocol DSLAM DSL Access Multiplexer IPC Inter process communication E2E End to end ISG Intelligent Services Gateway ECMP equal cost multipath iSPF incremental Shortest Path First EEM Embedded Event Manager ISSU in service software upgrade EOAM Ethernet OAM IWF Interworking function
For Your Reference
4
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Glossary (Cont.) L2TP Layer 2 transport protocol NIC network interface card LAC L2TP access concentrator Nr receive sequence number LACP Link aggregation control Protocol Ns send sequence number LAN Local Area Network NSF non stop forwarding LC Linecard NSR non stop routing LDP label Distribution Protocol NVRAM non volatile random access memory LFA loop free alternate OAM operations, administration and maintenance LI Lawful Intercept OCE Object Chain Element LMI Local management interface OIR online insertion and removal LNS L2TP network Server OS operating system LOS Loss of signal PADR PPP active discovery LSDB link state database PE provider edge LSP label switched path PIC prefix independent convergence LTE long term evolution PIM protocol independent multicast MC LAG multi chassis link aggregation PPP Point to point protocol mcast multicast PS power supply MD5 message Digest algorithm 5 PSN Packet Switched Network MFIB multicast forwarding information base PTA PPP termination and aggregation MLD multicast listener discovery PVRSTP Per VLAN rapid spanning tree MME mobile management entity PW pseudowire MoFRR Multicast Only fast reroute QFP Quantum flow Processor MPLS Multiprotocol label switching RADIUS remote authentication dial in user service MRIB multicast routing information base RF redundancy facility MSC mobile switching center RMA Return material authorization MSPP Multi-service provisioning platform RNC radio network controller MST Minimum spanning tree RP route processor MTBF mean time between failures RPR route processor redundancy MTSO mobile telephone switching office RSP route switch processor MTTR mean time to repair RSVP resource reservation protocol NAT network address translation SAA service assurance agent
For Your Reference
5
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Glossary (Cont.) SBC session border controller SBY standby SGW SAE gateway SIP Session initiation protocol SLA service level assurance SLB server loadbalancing SP service Provider SPA Shared port adapter SPF shortest path first SRLG shared risk link group SSH secure shell SSO stateful switchover STP spanning tree protocol SW software T&C terms & conditions TCAM ternary content addressable memory TE traffic engineering TR Traceroute UC unified communications uRPF unicast reverse path forwarding VAI virtual access interface VC Virtual Circuit VCCV VC connection verification VIP virtual IP VLAN virtual LAN VMAC virtual MAC VPN virtual private network VRF virtual routing and forwarding table VRRP virtual router redundancy protocol WAN wide area network
For Your Reference
6
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Agenda
Motivation for High Availability in SP Aggregation Networks
Network Level High Availability
System High Availability
Service High Availability
Stateful Inter-chassis Redundancy
Case Studies
Summary and Conclusions
7
Motivation for High-Availability in
the IP Edge and Aggregation
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
High Availability and Service Level Agreements
Many SPs specify their SLAs in the T&Cs
Important characteristic of both business and residential services
Historically given for Core network, but expanding to end-end SLAs
Metrics
Service Availability (averaged over time)
Mean time to repair (MTTR)
Packet Loss / Delay / Jitter
Examples
AT&T
Sprint
Verizon Business
BT
Level 3
9
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
What Is High Availability?
Availability DPM Downtime per Year (24x365)
99.000% 10000 3 Days 15 Hours 36 Minutes
99.500% 5000 1 Day 19 Hours 48 Minutes
99.900% 1000 8 Hours 46 Minutes
99.950% 500 4 Hours 23 Minutes
99.990% 100
53 Minutes
99.999% 10 5 Minutes
99.9999% 1 30 Seconds
Reactive
Proactive
“High Availability”
Predictive
Two ways to state availability of a network:
Percentage Method
DPM Method = Defects per Million (Hours of Running Time)
For Your Reference
10
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Availability Definitions
MTBF is Mean Time Between Failure
When does it fail?
MTTR is Mean Time To Repair
How long does it take to fix?
‘Uptime divided by the total time’ to create the
percentage time your network is operational
Availability = MTBF
MTBF + MTTR
For Your Reference
11
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Calculated vs. Measured Availability
Calculated Availability based on:
Network design
Component MTBF and MTTR
different underlying models, simulations
Cisco uses Industry standards to compute Hardware MTBF
Basic Availability Calculation Formulae:
Measured Availability based on:
ICMP Reachability (E2E, Device)
Cisco Service Assurance Agent (SAA)
Trouble Ticket / Outage Log Analysis
Observed Method: Shipping/RMA Method
= A1 ´A2 ´....´AN
=1- (1-A1)´....´(1-AN )
=
= N
k K
A 1
Series A
=
- - = N
K k
A 1
Parallel ) 1 ( 1 A
For Your Reference
12
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Reduction of MTTR
Stateful inter-chassis redundancy allows for additional
resilience against
System Failures
Interface Failures
Issue is not really MTBF of hardware modules, but
rather
Line failures / optical path failures
Interface failures
Power outages
Goal of stateful inter-chassis redundancy is sub-
second failover with state preservation for applications
Product ID MTBF (hrs)
ASR1000-RP2 380532
ASR1000-ESP20 335317
ASR1000-SIP10 287549
ASR1006 1986649
ASR1006-PWR-AC 570776
ASR1006-PWR-DC 357781
ASR1000-SIP40 283225
ASR1000-ESP40 118790
SPA-8X1GE-V2 482023
SPA-1X10GE-L-V2 411892
For Your Reference
13
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Device Availability Calculation
IOS Chassis and Backplane
P/S
P/S
IOS = = 0.999997 30.000 30.000 + 0.1
P/S = = 0.999995 750.000 750.000 + 4
System Availability = 0.999997 * ....*0.999983*(1-(1-0.999995)2) = 0.999961 = 99.9961%
IF1
CISCO7206-VXR
PWR-7200-AC
PA-E3 PA-POS-OC3
IF2 CPU
NPE-400
BB = = 0.999983 460.000 460.000 + 8
CPU = = 0.999992
490.000 490.000 + 4
12.0 GD Cisco 7206
IF1 = = 0.999996 1.120.000
1.120.000 + 4
IF2 = = 0.999993 600.000
600.000 + 4
Calculated MTBF Values from Cisco Database
For Your Reference
14
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Availability of R1 and R2 in Series
= (0.999961*0.999961) = 0.99992175
1
Network Availability Calculation
Router R1,R2,R3, R4: 0.999961
Availability of Parallel Network Path (R1-R4)
= 1 - ((1-0.999921)(1 - 0.999921)) = 0.999999994
2
Network Availability = 99.9999%
Only Based on Device Availability Values
3
R1
R4 R3
R2
but not considered: -Links (WAN, LAN) -Computer NICs -Computer OS -Computer Applications
For Your Reference
15
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Cisco High-Availability Focus
• increase MTBF using resilient HW/SW
• minimize MTTR for system failures (RP, LCs and SW)
• Mitigate planned outages by providing hitless software upgrades
System Level Resilience
• in the core and where redundant paths exist
• Deliver features for fast network convergence, protection & restoration
Network Level Resilience
• Embed intelligent event management for proactive maintenance
• Automation and configuration management to reduce human errors
Embedded Management
and Automation
16
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Cisco HA Feature Toolbox: System Level
RPR, SSO
NSF, NSR
SSO Multirouter APS
Stateful NAT/IPSec/Firewall/SLB stateful failover within single chassis
MPLS HA (L3VPN, L2VPN, InterAS, CSC, TE, FRR)
IOS / IOS XR / IOS XE ISSU, dual IOS XE
Service Provider Core
Service Provider Edge
Service Provider Aggregation
Access Layer
Data Center Building Block
Internet
17
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Network-Level Resiliency
Network Design Resiliency Dual-homing
APS, GEC, MC-LAG
Event Dampening
Fast Convergence iSPF Optimization (OSPF, IS-IS)
BGP Optimization
Fast BGP Convergence
Graceful Restart (MBGP, OSPF, RSVP, LDP)
EMCP, Anycast, dual RR
VRRP/HSRP/GLBP/SLB/FWLB
MPLS High Availability
LDP Graceful Restart
MPLS/VPN NSF
BFD
MPLS FRR Path Protection
MoFRR
IP FRR
Pseudowire Redundancy
Spanning Tree (MST, PVRSTP...)
‒..................
Internet
DC
Access
1st Level Aggregation
2nd Level Aggregation
IP Edge
SP Core
18
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Cost of High Availability
Designing a network for higher Service Availability
comes at a cost
Redundant Network Elements
Redundant Links
Redundant System Components (route processors,
forwarding processors, power supplies, etc.)
Operational costs
Lower steady-state Utilization levels
Increased configuration and management
Tighter maintenance windows
Availability
Cost ($)
0% 100%
19
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Cost of High-Availability: Example
Large SP Network
Residential Services (3-Play)
10M Subscribers
1.25 Mbps / subscriber
Up to 96% increased CAPEX
for full redundancy!
Opex increased due to higher number of
network elements
Edge MPLS Aggregation Access Residential Residential
Core
Subscribers AN Agg1 Agg2 Agg3 BNG P
Locations 200,000 4000 500 74 74 74
System Type Generic ASR 9000 ASR 9000 ASR 9000 ASR 1013 CRS-3
Redundancy Scheme Total Cost $M
Chassis Costs $M
Interface Costs (SPA, SFPs), $M
Number of nodes
No redundancy $1,232 $529 $704 4658
Access NW uplink redundancy (Agg1, Agg2, Agg3)
$1,250 $529 $721 4658
AN uplink redundancy $1,563 $531 $1,032 4680
Access Network node redundancy (Agg1, Agg2, Agg3)
$2,423 $1,044 $1,379 9222
Edge link redundancy $2,425 $1,044 $1,381 9222
Edge Node redundancy $2,437 $1,056 $1,381 9296 Values for AN, Agg1, Agg2, Agg3 and Edge nodes only (No Pp-routers). Cumulative redundancy Schemes, GPL
20
Network High Availability
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
End-to-End Service HA Solution Set
0
0 – System-level HA (Baseline) RSP failover: 0 packet loss All L3 protocols are NSF capable NSR: OSPF, ISIS, BGP Routing timers and protocol configs are optimized by default
1
1 – Distributed BFD Rapid failure detection 15ms timer High scale
2 – BGP PIC Prefix Independent Convergence Fast Core/edge failure convergence
2
3
3 – TE/FRR Link/Node/Path Protection sub-50msec network convergence
4
4 – ECMP 32-way IGP/LDP, 8-way BGP Dynamic ECMP
5 – LACP Ethernet bundle With Full L3 support
6
7
7 – HSRP/VRRP Excellent Scale 100ms timer BFD Integration
5
6 – MPLS IP/TE FRR IP FRR w/ OSPF/ISIS sub-50msec network
convergence
SP Core Aggregation
For Your Reference
22
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
HA Network Map
AN <-> AGG <-> Edge <->
L4-7
App
Failure Detection Keepalives
Recovery Stateful App Redundancy
L3 Failure Detection BFD Keepalives BFD, Keepalives
Recovery MPLS TE FRR, IP Event Dampening, Fast Convergence, IP FRR, NSR / NSF
HSRP /
VRRP/GLBP/SLB/FWLB
Multicast HA
ECMP, iSPF, BGP PIC Core /
Edge, IP / MPLS FRR, LNS
Load sharing / Anycast / Dual
RR, Fast Hello
L2 Failure Detection
EOAM, (VCCV) Keepalives VCCV, EOAM,
MPLS Ping / TR
Keepalives EOAM, MPLS Ping / TR
Recovery GEC / APS / MC-LAG PW redundancy
Bridge Domains
GEC / APS / MC-
LAG
PPP / FR / ATM / HDLC /
GE SSO
GEC, APS, MC-LAG
L0/1 Failure Detection Interrupts Loss of Signal Interrupts Loss of Signal Interrupts
Recovery Module
Redundancy
Path diversity / dual
homing
Module Redundancy Path diversity / dual
homing
Module Redundancy
Aggregation Core Edge Access
Access Domain
23
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
HA Network Map
AN <-> AGG <-> Edge <->
L4-7
App
Failure Detection Keepalives
Recovery Stateful App Redundancy
L3 Failure Detection BFD Keepalives BFD, Keepalives
Recovery MPLS TE FRR, IP Event Dampening, Fast Convergence, IP FRR, NSR / NSF
HSRP /
VRRP/GLBP/SLB/FWLB
Multicast HA
ECMP, iSPF, BGP PIC Core /
Edge, IP / MPLS FRR, LNS
Load sharing / Anycast / Dual
RR, Fast Hello
L2 Failure Detection
EOAM, (VCCV) Keepalives VCCV, EOAM,
MPLS Ping / TR
Keepalives EOAM, MPLS Ping / TR
Recovery GEC / APS / MC-LAG PW redundancy
Bridge Domains
GEC / APS / MC-
LAG
PPP / FR / ATM / HDLC /
GE SSO
GEC, APS, MC-LAG
L0/1 Failure Detection Interrupts Loss of Signal Interrupts Loss of Signal Interrupts
Recovery Module
Redundancy
Path diversity / dual
homing
Module Redundancy Path diversity / dual
homing
Module Redundancy
Aggregation Core Edge Access
Access Domain
For Your Reference
24
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Link Failure Detection Mechanisms
Loss of Signal (LOS) key to link failure detection
O(ms) with interrupt driven LoS detection on ASR 9000
Carrier Delay may be used to become resilient to link flaps
BFD
Ethernet OAM
MPLS OAM
Media type CC CP CC DP Loopback Performance Traceroute
Ethernet Last Mile IEEE 802.1ah - - -
Ethernet Provider Bridge
IEEE 802.1ag (MAC: Broadcast Domain)
MPLS LDP LDP Hello BFD, Y.1713, Y.1711
LSP Ping -
LSP TR MPLS TE RSVP Hello -
MPLS PW LDP Hello BFD, Y.1711 VCCV Ping - -
IPv4 IGP/BGP
Hello BFD IP Ping - IP TR
For Your Reference
25
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
E-LMI - Provides protocol and mechanisms used for:
Notification of EVC addition, deletion or status to CE
Communication of UNI and EVC attributes to CE
CE auto-configuration
Notification of Remote UNI name and status to CE
IEEE 802.3ah
OAM Discovery
Link Monitoring
Fault Signaling
Remote MIB Variable Retrieval
Remote Loopback
IEEE 801.3ag (CFM)
Family of protocols that provides capabilities to detect, verify, isolate and report end-to-end ethernet connectivity faults
Protocols (Continuity Check, Loopback and Linktrace) used for Fault Management activities
Ethernet OAM Overview BRKSPG-2202
For Your Reference
26
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Ethernet OAM Overview
802.1ag
E-LMI
802.3ah 802.3ah 802.3ah 802.3ah 802.3ah 802.3ah
Core
E-LMI
Ethernet LMI: Automated configuration of CE based on EVCs and bandwidth profiles
L2 connectivity management
IEEE 802.3ah: When applicable, physical connectivity management between devices.
IEEE 802.1ag: Connectivity Fault Management (CFM)
Uses Domains to contain OAM flows and bound OAM responsibilities
Provides per EVC connectivity management and fault isolation
Three types of packets: Continuity Check, L2 Ping, L2 Traceroute
BRKSPG-2202
27
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
IEEE 802.1ag CFM Concepts
Nested Maintenance Domains (MDs)
break up the responsibilities for network administration of a given end-to-end service
Defined by operational boundaries
Nest & touch, but do not intersect
Maintenance Associations (MAs)
monitor service instances under a given MD
Defined by set of MEPs at the edge of a domain
Identified by {MA Name + MD ID}
Maintenance Association End Points (MEPs)
generate and respond to CFM PDUs
Define boundaries of MD
Initiate & respond to CFM PDUs
Per-Maintenance Association multicast “heart-beat” messages
Carries status of port on which MEP is configured
Uni-directional (no response required)
Transmitted at a configurable periodic interval by MEPs
Catalogued by MIPs at the same MD-Level and service, Terminated by remote MEPs in the same MA
Maintenance Domain
Maintenance Association
Maintenance Association End Points
28
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
ITU-T Y.1731 Overview
OAM Functions for Fault Management
Ethernet Continuity Check (ETH-CC) (Y.1731 adds unicast CCM)
Ethernet Loopback (ETH-LB) (Y.1731 adds multicast LBM)
Ethernet Linktrace (ETH-LT)
Ethernet Remote Defect Indication (ETH-RDI)
Ethernet Alarm Indication Signal (ETH-AIS)
Ethernet Locked Signal (ETH-LCK)
In addition: ETH-TEST, ETH-APS, ETH-MCC, ETH-EXP, ETH-VSP
OAM Functions for Performance Management
Frame Loss Measurement (ETH-LM)
Frame Delay Measurement (ETH-DM)
Covered by IEEE 802.1ag
29
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Virtual Circuit Connection Verification (VCCV)
Overview checks connectivity between egress and ingress PEs
VCCV allows sending control packets in band of pseudowires (PW)
Signaling component: communicate VCCV capabilities as part of VC label
Switching component: cause the PW payload to be treated as a control packet
VCCV capability is negotiated when the AToM tunnel is brought up
depends on the LDP peer and the VC type
Both endpoints must have the same capabilities
marks the payload as control packet for switching purpose; packet follows the PW data path
Control packets sent over the AToM tunnels are intercepted by the egress PE
Type 1
(in-band vccv)
To signal in-band VCCV [RFC4385] using PW ID from PW Control Word
Type 2
(out-of-band VCCV)
Signal out-of-band VCCV inserting MPLS router alert label between tunnel and PW Labels
Type 3 (TTL expiry) Manipulate and Signal TTL exhaust (TTL == 1) for multiple switching point PEs
30
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
AC AC PE2 PE1 LDP Notification Message PW Status TLV
PW Status Code
MPLS Pseudowire Status Signaling Procedure
PW Status Signaling method selected if supported by both peers.
PEs exchange label mapping messages upon PW configuration.
Simple Label Withdraw status method will be used if one of the peers doesn’t support PW Status Signaling.
PW label won’t be withdrawn unless AC is administratively down or the PW configuration is deleted.
PW state set to “down” if the Label mapping is not available.
Capability is on by default.
31
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Virtual Circuit Connection Verification (VCCV)
Multiple Packet Switched Network (PSN) Tunnel Types MPLS, IPSEC, L2TP, GRE,…
Motivation One tunnel can serve many pseudo-wires.
MPLS LSP ping is sufficient to monitor the PSN tunnel (PE-PE connectivity), but not
Virtual Circuits (VCs) inside of tunnel.
CE1 CE2 PE1 PE2
PSN Tunnel PW1
PW2
Emulated Service
Pseudo Wire
Native Service Native Service
For Your Reference
32
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BFD Protocol Overview
Accelerates convergence by running fast keepalives in a consistent, standardized mechanism across routing protocols
Lightweight hello protocol
Neighbors exchange hello packets at negotiated regular intervals
Configurable transmit and receive time intervals
Unicast packets, even on shared media
No discovery mechanism
BFD sessions are established by the clients e.g. OSPF, IS-IS, EIGRP, BGP, …
Client hello packets transmitted independently
BFD Control Packets
OSPF
BGP
EIGRP
IS-IS
BFD
OSPF
BGP
EIGRP
IS-IS BFD
33
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BFD Details
Session established between two peers
Timers are negotiated
Hello packets similar to IGP control packets
Does NOT react to failures itself -> notifies clients
Async mode (no echo): periodic control packets sent
Neighbour declared dead if no pkt is received for <interval *
multiplier> period
Session established using async control session
Echo mode: echo packets sent at negotiated rate,
used for failure detection
Control packets sent at low rate
Scalability: between 500 and 4000 sessions
Scalability depends on timer settings (50ms – 1sec)
green is alive
orange is alive
orange is alive
green is alive
Async Mode
Echo Mode
For Your Reference
34
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Network Convergence — Why It Takes So Long
Detection of Link layer failure ms
ms
10’s of ms
10’s of ms
100’s of ms
10’s of ms
Report failure to Route Controller
Generate and flood an LSP
Trigger and Compute an SPF
Communicate new FIB entries to linecards
Install new FIB entries into linecard HW path
Bottleneck
35
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Network Convergence — Why It Takes So Long
Detection of Link layer failure ms
ms
10’s of ms
10’s of ms
100’s of ms
10’s of ms
Report failure to Route Controller
Generate and flood an LSP
Trigger and Compute an SPF
Communicate new FIB entries to linecards
Install new FIB entries into linecard HW path
Optimize IGP
Convergence
BGP PIC
Optimize LDP & BGP
Convergence
36
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Hierarchical CEF
Optimizes the data plane for sub-second convergence
CEF Data Structure Enhancements
Solves the FIB Download Convergence Bottleneck LSP and Prefix Independent
Optimizes FIB
Hierarchical CEF Technologies
MPLS-FRR
IP-FRR
BGP PIC Core
BGP PIC Edge
Non-Hierarchical CEF Technologies
MPLS Path Protection
Default – IP to MPLSDefault – IP to MPLS
One CEF Update Message per
prefix
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
Adjacency OCE - Interface
Adjacency OCE - Interface
Adjacency OCE - Interface
X
Failure
Repair
MPLS Label OCE
MPLS Label OCE
MPLS Label OCE
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
MPLS Label OCE
MPLS Label OCE
MPLS Label OCE
PIC Core – IP to MPLSPIC Core – IP to MPLS
One CEF Update Message for
Multiple Prefixes
Adjacency OCE - Interface
Adjacency OCE - Interface
Adjacency OCE - Interface
X
Failure
Repair
Load Balance OCE
Load Balance OCE
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
MPLS Label OCE
MPLS Label OCE
MPLS Label OCE
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
MPLS Label OCE
MPLS Label OCE
MPLS Label OCE
37
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
IP/MPLS Aggregation
MPLS FRR 50 ms Convergence
Key Features
Fast Convergence for Link and Node Failures
Supported Across all Network Topologies
MPLS-TE Traffic Management
SRLG
BW Reservation
Per Tunnel Traffic Statistics
Caveats
Requires MPLS and MPLS-TE
No Protection for Ingress or Egress Tunnel Failures
Requires Pre-Computed Backup Paths
Requires “(n-1)!” Tunnels for Full Protection
Applicability Protecting Links in the aggregation network
Tunnel LSP
VC LSPs
Link Failure
FRR LSP
BRKRST-3363
38
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
MPLS-FRR — CEF
Typical FIB Programming Rate - ~5000 – 10,000 CEF Updates per second
MPLS-FRR – IP and MPLS
One CEF Update Message
Adjacency OCE - Interface X
Failure
Repair
Midchain OCE
Loadbalance OCE
Loadbalance OCE
Loadbalance OCE Adjacency OCE - Interface
FRR OCE Label OCE
Label OCE
Midchain OCE
Loadbalance OCE
Loadbalance OCE
Loadbalance OCE Adjacency OCE - Interface
FRR OCE Label OCE
Label OCE
IP & MPLS CEF
IP & MPLS CEF
Pre-computed MPLS-FRR Backup Path
For Your Reference
39
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
IP/MPLS Aggregation
Tunnel LSP
VC LSPs
Protect Tunnel LSP
Link Failure
MPLS Path Protection
Key Features
Optimized for Ring Topologies
Utilizes Pre-Signaled Backup Tunnel
MPLS-TE Traffic Management
SRLG
BW Reservation
Per Tunnel Traffic Statistics
Caveats
Requires MPLS and MPLS-TE
No Protection for Ingress or Egress Tunnel Failures
Convergence Dependant on IGP Prefixes and L2VPN LSPs Under
Protection
Applicability
Protecting Ring Topologies
40
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
MPLS Path Protection — CEF
Typical FIB Programming Rate - ~5000 – 10,000 CEF Updates per second
MPLS-FRR – IP and MPLS
Adjacency OCE - Interface X
Failure
Repair
Midchain OCE
Loadbalance OCE
Loadbalance OCE
Loadbalance OCE
MPLS-TE Path Protect Tunnel
Label OCE IP & MPLS CEF
Adjacency OCE - Interface Midchain OCE Label OCE
Loadbalance OCE
Loadbalance OCE
Loadbalance OCE
IP & MPLS CEF
Adjacency OCE - Interface Midchain OCE Label OCE
One CEF Update Message per IGP Prefix and L2VPN LSP!
For Your Reference
41
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
IP FRR-LFA: 50 ms Convergence
Key Features
50 msec Convergence for Link and Node Failures
Works for MPLS and IP Only Environments
Simple
Automatic configuration of “Loop Free Alternate Paths” via OSPF or ISIS
No Tunnels
Caveats
Requires a “Loop Free Path” for Protection
No Bandwidth Reservation
No Support for SRLG
New Feature
Applicability
Strong Solution for Deployments with Cost Effective Bandwidth
Loop Free Path
R1 R2
R3 R4
R5
Link Failure
No Convergence Required on
Routers R2, R3, R4 and R5 to
Maintain Green Traffic Flow!
BRKRST-3363
42
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Tight SLA Protection with IPFRR
IPFRR Loop Free Alternate (LFA) - Principle
of Operation
If A finds an alternate path to B
Then this alternate path is valid for any destinations that A normally routes via B
IPFRR LFA – Properties and Benefits
Automated – No additional setup
No IETF protocol change - all the needed info is already in classical LSDB
Incremental deployment
No inter-operability testing
<50msec, prefix-independent
Applicable to MPLS LDP networks
IPFRR LFA – Deployment Dependency
IPFRR LFA coverage depends on the network topology
Two-plane network topologies are most “friendly” for IPFRR LFA deployments
Topology analysis required to assess IPFRR LFA efficiency
IPFRR LFA – Principle of Operation
Two-plane Network Topology
LSDB = Link State DataBase LDP = Label Discovery Protocol
For Your Reference
43
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
IP FRR-LFA — CEF Enhancement
Typical FIB Programming Rate - ~5000 – 10,000 CEF Updates per second
IP FRR-LFA – IP to MPLS
One CEF Update Message for Multiple Prefixes
Adjacency OCE - Interface X
Failure
Repair
Load Balance OCE IP Prefix FIB Entry
IP Prefix FIB Entry
IP Prefix FIB Entry
MPLS Label OCE
Adjacency OCE - Interface
IP-FRR OCE Load Balance OCE MPLS Label OCE
Load Balance OCE MPLS Label OCE
Load Balance OCE IP Prefix FIB Entry
IP Prefix FIB Entry
IP Prefix FIB Entry
MPLS Label OCE
Adjacency OCE - Interface
IP-FRR OCE Load Balance OCE MPLS Label OCE
Load Balance OCE MPLS Label OCE
Cleanup After Repair – Assuming No Available Loop Free Path
Load Balance OCE IP Prefix FIB Entry
IP Prefix FIB Entry
IP Prefix FIB Entry
MPLS Label OCE
Load Balance OCE MPLS Label OCE
Load Balance OCE MPLS Label OCE
Adjacency OCE - Interface
Adjacency OCE - Interface
Adjacency OCE - Interface
Pre-computed backup Path
For Your Reference
44
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Non Stop Forwarding (NSF)
Routers to maintain forwarding state when communication between them is lost
Routing sessions are established with NSF aware peers. Upon HA event, neighboring peers maintain forwarding until routing sessions are reestablished.
Copy of FIB maintained on secondary and used on failure for continuously traffic flow.
Requires neighboring routers to be NSF aware.
Traffic is forwarded continuously
NSF Aware eBGP
CE1 PE1 P1
NSF Aware IGP and LDP
45
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge
PE1
VPN1 VPN1
PE2
CE1 CE2
PE3
Link Failure
Optimizes BGP Convergence for BGP Next-Hop Change
PE to CE Link Failures
PE Node Failures
CE Node Failures
Applicability
PE Routers
Requires “bgp advertise-best-external” to enable
BRKRST-3363
46
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-CE Link Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
Traffic flow due to BGP best path
47
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-CE Link Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
Traffic flow due to BGP best path
The
BG
P
pre
-cal
cula
ted
B
acku
p P
ath
48
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-CE Link Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
Traffic flow due to BGP best path
The
BG
P
pre
-cal
cula
ted
B
acku
p P
ath
The best BGP path to CE1 is now through
PE2
PE-CE link Failure
Detects that link is down and CEF layer
will switch to pre-computed backup path
49
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-CE Link Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
The best BGP path to CE1 is now through
PE2
PE-CE link Failure
50
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-CE Link Protection
PE1 and PE2 precomputes bgp backup paths using bgp best-external approach
When primary link PE1 - CE1 fails:
PE1 holds on to the bgp local labels and re-routes CE1’s traffic to PE2 using labels advertised by PE2
PE1 uses fixed timer to clean up stale local labels
PE3 is expected to converge to start using PE2 as the BGP nexthop and IGP label for PE2 to send traffic from CE2 to CE1
CE2
PE1
PE2
CE1 PE3
MPLS-VPN
On Backup PE: router bgp 100 address-family ipv4 vrf red bgp advertise-best-external bgp additional-paths install
On Primary PE: router bgp 100 address-family ipv4 vrf red bgp additional-paths install bgp advertise-best-external
For Your Reference
51
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-Node Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
Traffic flow due to BGP best path
Relies on BGP
Add-path
The best BGP path
to CE1 is through
PE1
52
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-Node Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
Traffic flow due to BGP best path
The best BGP path
to CE1 is through
PE1
53
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-Node Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
Detects that PE1 is
down and CEF layer
will switch to pre-
computed backup path
IGP signals router
is dead
PE-CE node
Failure
54
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE-Node Protection
PE2
PE1
CE1 RR1
RR2
PE3
CE2
next BGP next-hop
scan the path through
PE2 will become the
best Path PE-CE node
Failure
55
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge PE Node Protection
PE1, PE2 and PE3 precomputes bgp backup
When node PE1 fails:
IGP notification on PE3 invalidates active path
PE3 switches to backup path
PE3 is expected to converge to start using PE2 as the BGP nexthop and IGP label for PE2 to send traffic from CE2 to CE1
CE2
PE1
PE2
CE1 PE3 MPLS-VPN
On Primary PE: router bgp 100 address-family ipv4 vrf red bgp additional-paths install bgp advertise-best-external
On Backup PE: router bgp 100 address-family ipv4 vrf red bgp advertise-best-external bgp additional-paths install
On Ingress PE: router bgp 100 address-family ipv4 vrf red bgp additional-paths install
For Your Reference
56
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge Notes
Supported for IPv4/v6 and VPNv4/v6
Not supported for L2VPN and mVPN address families
Failures detected using BFD or IGP
57
For Your Reference
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP PIC Edge — CEF
Typical FIB Programming Rate - ~5000 – 10,000 CEF Updates per second
PIC Edge – IP to MPLS
One CEF Update Message for Multiple Prefixes
Adjacency OCE - Interface X
Failure
Repair
Load Balance OCE
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
MPLS Label OCE X Adjacency OCE - Interface MPLS Label OCE
Pre-Computed Backup Path
Load Balance OCE
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry Adjacency OCE - Interface MPLS Label OCE
Adjacency OCE - Interface
Cleanup After Repair
Load Balance OCE
BGP Prefix FIB Entry
BGP Prefix FIB Entry
BGP Prefix FIB Entry
MPLS Label OCE
Adjacency OCE - Interface MPLS Label OCE Load Balance OCE
For Your Reference
58
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
VPN1 site1
VPN1 Site2
1
3
BGP PIC core – when IGP path to BGP Next-Hop changes 1. Examples: PE-P or P-P link failure, P node failure Sub-second convergence (prefix independent) vs. multiple seconds convergence (prefix and hardware dependent) Enabled by default since IOS XE 2.5.0 (cef table output-chain build favor convergence-speed)
BGP PIC edge – When BGP Next-hop changes 2. when remote PE node fails or no longer reachable. 3. when PE-CE link fails. Immediate to sub-second convergence (prefix independent) vs. multiple seconds convergence (prefix and hardware dependent)
BGP PIC Edge vs. BGP PIC Core
PE3
PE1
PE2 2
CE1
CE2
For Your Reference
59
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
IP/MPLS High-Availabilty Options: Scorecard
Network Infrastructure- all transit links and nodes
IGP Fast Convergence (IGP FC)
Broke the barrier of <200msec restoration time
Covers all faults, including multiple failures
IP/MPLS Fast ReRoute Loop Free Alternate (IPFRR LFA)
Provides local protection (link, node) with <50msec recovery
Tool to improve on IGP FC for most topologies (triangle, square, mesh)
MPLS TE Fast ReRoute (TE FRR)
Provides local protection (link, node, path) with <50msec recovery
Service Edge- edge node and access links
BGP Prefix Independent Convergence (BGP PIC)
IP/IPVPN scale independent recovery in line with IGP FC and FRR
Applicable to all BGP based services (IPv4, IPv6, VPNv4, VPNv6)
Fault Coverage
Operational Simplicity
Recovery Time
O(x00ms)
<50ms
<50ms
O(x00ms)
Feasible to deliver very tight E2E Service Availability SLAs without increasing operational complexity
For Your Reference
60
System High Availability
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
System High-Availability with Hardware
Redundancy Redundant hardware components
Power Supplies
Route Processors
Forwarding Processors
Switching Matrix
SPA Interface Cards
Interface Redundancy typically achieved using IEEE 802.3ad / LACP or APS
Hardware Redundancy needs to be complemented by Software redundancy Features
Cisco Platforms supporting hardware redundancy
CRS-3 ASR 9000 ASR 5000 ASR 1000 Cisco 12000 Cisco 7600 62
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Distributed Forwarding Plane for Performance
Up to Eight Linecards
(Autonomous Forwarding)
Distributed IOS®XR based Control Plane for Scale
Dual Route Switch Processors (RSPs)
Dual-Core CPU on Each Linecard
Active/Active Switch Fabric for HA
Non-blocking Memory-less Fabric
Service Intelligence with Hi / Lo Priorities,
Unicast & Multicast Recognition, and VoQ’s
Redundant EOBC, Fan Trays, Power supplies
ASR 9000 System Architecture
MAC
FIC
NPU
NPU
NPU
NPU MAC
MAC
FIC
NPU
NPU
NPU
NPU MAC
FIC
CPU BITS/DTI
RSP(s) Line Card (40G)
FIC
CPU BITS/DTI
MAC
FIC
NPU
NPU
NPU
NPU MAC
CPU
MAC
FIC
NPU
NPU
NPU
NPU MAC
CPU
63
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Control Plane Data Plane
Example: ASR 1000 System Redundancy
SPA SPA
IOCP SPA
Agg.
…
Interconn.
SPA SPA
IOCP SPA
Agg.
…
Interconn.
SPA SPA
IOCP SPA
Agg.
…
Interconn.
Route Processor (Standby)
Route Processor
(active)
Embedded Services Processor (Standby)
FECP
Interconn.
QFP subsys-tem Crypto
assist
Embedded Services Processor
(active)
FECP
Interconn.
QFP subsys-tem Crypto
assist
Passive Midplane
RP RP
GE, 1Gbps I2C SPA Control SPA Bus
Route Processor (standby)
RP
Interconn.
Embedded Services Processor
(active)
FECP
Interconn.
QFP subsys-tem Crypto
assist
Embedded Services Processor (standby)
FECP
Interconn.
QFP subsystem Crypto
assist
SPA SPA
IOCP SPA
Agg.
…
Interconn.
SPA SPA
IOCP SPA
Agg.
…
Interconn.
SPA SPA
IOCP SPA
Agg.
…
Interconn.
Passive Midplane
Route Processor
(active)
RP
Interconn.
SPA-SPI, 11.2Gbps Hypertransport, 10Gbps
ESI, (Enhanced Serdes) 11.5Gbps
64
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
ASR 1000 Software Architecture
ESP FECP
Interconn.
Crypto assist
RP CPU Chassis Mgr.
Forwarding Mgr.
Chassis Mgr.
Forwarding Mgr.
QFP Client / Driver
Interconn.
Interconn.
SIP
SPA SPA
IOCP
SPA Agg.
…
Interconn.
Kernel (incl. utilities)
Chassis Mgr. SPA driver
SPA driver
SPA driver
SPA driver
IOS
Kernel (incl. utilities)
Kernel (incl. utilities)
Kernel (incl. utilities)
Kernel (incl. utilities)
QFP subsys-tem
QFP code
Runs Control Plane Generates configurations Populates and maintains routing tables (RIB, FIB…)
Provides abstraction layer between hardware and IOS (manages ESP
redundancy) Maintains copy of FIB and interface list Communicates FIB status to active & standby ESP (or bulk-download
state info in case of restart)
Maintains copy of FIBs Programs QFP forwarding plane and QFP DRAM Statistics collection and communication to RP
Communicates with Forwarding manager on RP Provides interface to QFP Client / Driver
Implements forwarding plane Programs PPEs with forwarding information
For Your Reference
65
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
RPact
FMRP
QFP Client
RPsby
SPAs
ASR 1006 High Availability Infrastructure
HA operates in a similar manner to other protocols on the ASR 1000
Reliable IPC transport used for synchronization
RF RF
IPC Message Qs
IDB State Update Msg IDB State Update Msg
IOSact IOSsby
I P C
I P C
CF CF Interconnect Used for IPC and Check-pointing
Non-HA-Aware Application
Non-HA-Aware Application
Driver/Media Layer
Mcast
CEF
Config
…
Driver/Media Layer
Mcast
CEF
Config
… MLD MLD
IPC Message Qs
ESPsby ESPact
QFP Client
FMRP
FIB MFIB FIB MFIB
FMESP FMESP
IDB RIB RT MRIB IDB RIB RT MRIB
FIB MFIB FIB MFIB
66
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Which Events Trigger Failovers?
The following events may trigger failovers on the RP/ESP
1. Hardware component failures
2. Software component failures
3. Online Insertion and Removal (OIR)
4. CLI-initiated failover (e.g. reload command, force-switchover command)
67
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
1. Failover Triggers: Hardware Failures
What hardware failures?
a. CPUs: RP-CPU, QFP, FECP, IOCP, interconnect CPU, I2C Mux, ESP Crypto Chip,
b. Memory: NVRAM, TCAM, Bootflash, RP SDRAM, FECP SDRAM, resource DRAM, Packet
buffer DRAM, particle length DRAM, IOCP SDRAM, …
c. Interconnects: ESI Links, I2C links, EOBC Links, SPA-SPI bus, local RP bus, local FP bus …
d. Other: heat-sinks, …
Detected using
CPLD interrupts / register bits within O(ms) controlled by CMRP
Watchdog timers: low level watchdogs running in O(min) that can initiate a reset (e.g. RP)
JTAG: RP can program CPLD on other modules. Test interconnects and other boards (primarily
for RMAd hardware)
Interrupts generated by hardware failures initiate fail-over events
Hardware failures are typically fatal such that modules need to be
replaced!
SIP
SPA SPA
IOCP
SPA
Agg.
…
Interconn.
ESP FECP
Interconn. QFP subsys-tem
Crypto assist
RP CPU
Interconn.
Interconn.
68
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
2. Failover Triggers: Software Failures
SIP
SPA SPA
IOCP
SPA
Agg.
…
Interconn.
ESP FECP
Interconn. QFP subsys-tem
Crypto assist
RP CPU
IOS
Chassis Mgr.
Forwarding Mgr.
Chassis Mgr.
Forwarding Mgr. QFP Client / Driver
Interconn.
Chassis Mgr. SPA driver
SPA driver
SPA driver
SPA driver
Interconn.
QFP code
IOS
Kernel (incl. utilities)
Kernel (incl. utilities)
Kernel (incl. utilities)
What Software Failures?
a. Kernel: Linux on RP / ESP / SIP
b. Middleware: Chassis Manager (CM), Forwarding Manager (FM)
c. IOS
d. SPA drivers
Detected using
Kernel: the kernel supervises middleware or SPA driver processes (kmonitor()). It always knows if a process is healthy
IPC: between 2 IOS (and only for IOS)
Kernel will take the module down in a controlled manner
IOS, CMESP, CMSIP, FMESP, QFP Driver/Client, IMSIP are not re-startable!
Also setting register bits to initiate fail-over for ESP or RP
Note: some other processes are re-startable (CMRP, FMRP, SSH, Telnet…)
Kernel will try to re-start the processes in this case
69
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
SIP
CMSIP Kernel
RP (slot 1)
CMRP Kernel IOS
RPact Failover Procedure
ESP (slot 0) CMESP Kernel FMESP FMRP
RP (slot 0)
CMRP Kernel IOS FMRP
ESP (slot 1) CMESP Kernel FMESP
SIP
CMSIP Kernel
SIP
CMSIP Kernel
ACT ACT SBY SBY
Failure
Detect RPact
failure
ACT Restart New RPact information
Close ESI links (ESP)
Establish ESI links
State information
If not received in time, send restart message.
Update H/W component file system
ESI link status
For Your Reference
70
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Failover
Take-over control using checkpointed state Forwarding State information
Check updated state and discard old state
Check updated state and discard old state
Service recovered
Start kernel start
Start CM Start IOS
H/W initialization
Initialize EOBC
SIP
CMSIP Kernel
RP (slot 1)
CMRP Kernel IOS
RPact Failover Procedure (cont.)
ESP (slot 0) CMESP Kernel FMESP FMRP
RP (slot 0)
CMRP Kernel IOS FMRP
ESP (slot 1) CMESP Kernel FMESP
Start FM
SIP
CMSIP Kernel
SIP
CMSIP Kernel
Detect RPsby
SBY Forwarding State information ESI link status
Other RP information
Run Mastership
For Your Reference
71
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
SIP
CMSIP Kernel
RP (slot 1)
CMRP Kernel IOS
ESPact Failover Procedure
ESP (slot 0) CMESP Kernel FMESP FMRP
RP (slot 0)
CMRP Kernel IOS FMRP
ESP (slot 1) CMESP Kernel FMESP
SIP
CMSIP Kernel
SIP
CMSIP Kernel
ACT ACT SBY SBY
Failure Interrupt
ESI link status
Reconfigure ESI link w/ RPs
Disable ESI link w/ failed ESP
Change state of ESI link w/ new ESPact
ESI link status
Detect ESPact failure
State information of failed ESP
ACT Failed
Service Recovered with momentary packet loss
Resend state information
ACT ACT SBY
For Your Reference
72
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
SIP
CMSIP Kernel
RP (slot 1)
CMRP Kernel IOS
ESPact Failover Procedure (cont.)
ESP (slot 0) CMESP Kernel FMESP FMRP
RP (slot 0)
CMRP Kernel IOS FMRP
ESP (slot 1) CMESP Kernel FMESP
SIP
CMSIP Kernel
SIP
CMSIP Kernel
H/W initialization
Initialize EOBC
Restart
Wait for RPact
RPact information
Detect RPact
Activate ESI-link Download software packages
Start kernel
Start CM Register with CMRP
Other-ESP information (e.g. mastership)
SBY
Start FM
For Your Reference
73
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
IOS High Availability
During the initialization process IOS is loaded on both the RPact and Rpsby
IOS learns about any events Uses Redundancy and checkpointing facility
Redundancy facility (“when to synchronize”) A process to help in synchronization and coordination of switchovers (e.g. switchover events, switchover control and monitoring)
Clients of the RF maintain the databases that are synchronized using the RF
Examples: reliable internal data transfer service, Event notification mechanisms, logging, etc.
Checkpointing facility: (“how to synchronize”) Defines a set of APIs and transport for different SSO-aware features to copy state
Helps to synchronize state data in a consistent, repeatable and well-ordered manner
Keeps checkpointing state
Interfaces to RF
Line Card
RF RF
IPC Message Queues
IPC Message Queues
IDB State Update Msg IDB State Update Msg
Active RP Standby RP
I P C
I P C
CF CF
Interconnect Used for IPC and Check-pointing
Non-HA-Aware Application
Non-HA-Aware Application
Driver/Media Layer
PPP
CEF
L2TP
Config
…
Driver/Media Layer
PPP
CEF
L2TP
Config
…
IDB FIB IDB FIB
For Your Reference
74
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Cisco Software High-Availability Support
Stateful Switchover (SSO) support for features provides the synchronization of dynamic feature state between hardware modules
Configuration synchronization ensures that the running config is synchronized on the route processors
Dynamic State Preservation ASR 1000 ASR 9000
Connectivity Protocols FR, PPP, MLPPP, HDLC, 802.1Q, BFD (BGP, IS-IS, OSPF) BFD (OSPF, BGP, IS-IS, Static)
Routing & IP Services
RP, HSRP, IPv6 NDP, uRPF, SNMP, GLBP, VRRP, NSR (MP-iBGP,
eBGP), ISSU, GRE,
NSF (ISIS, OSPF, BGP), NSR (ISIS, OSPFv2,
OSPFv3, BGP)
Multicast IPv4 Multicast (IGMP), IPv6 Multicast (MLD, PIM-SSM, MLD Access
group), MoFRR
NSF Multicast, BFD for PIM, MoFRR
MPLS Protocols MPLS L3VPN, MPLS LDP , VRF-aware BFD,
Roadmap: NSR LDP, T-LDP
NSF (LDP, T-LDP, RSVP-TE)
NSR (LDP), BFD for MPLS FRR, VRF-aware BFD
Broadband PPPoE, L2TP (LAC, LNS), DHCPv4/v6, AAA, session state (virtual
templates), ISG, ANCP, LI
PPPoE (including nV)
Security SSO, Stateful Inter-chassis redundancy for FW / NAT Roadmap
SBC SSO Roadmap
For Your Reference
75
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Non Stop Routing (NSR)
Routers to maintain routing state and forwarding state when communication between them
is lost
Routing sessions are maintained between processors on a failure, allowing routing sessions
to stay up with Peer
Copy of FIB maintained on secondary and used on failure for continuously traffic flow
No need for neighboring routers to be NSF aware or capable. Can give high
reliability without upgrading CE.
Traffic is forwarded continuously
eBGP
CE1 PE1 P1
IGP and LDP
76
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
BGP NSR
Implemented by hardening code for
BGP RIB checkpointing
BGP TCP interaction
Only supported for IPv4 unicast, VPNv4 unicast Address families in Cisco IOS
Configuration router bgp <asn>
address-family ipv4 vrf RED
neighbor x.x.x.x ha-mode sso
No peer session flaps on VPNv4 CE when RP switches over
Route updates during RP switchover are announced to VPNv4 CE peers
NO delays which prevents data black-holes during RP switchover as in the case of graceful restart peers
77
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
OSPFv2 NSR
Provides the ability to perform hitless RP switchovers when OSPF is used as the routing protocol (Expect zero-traffic loss across such HA events)
To enable OSPFv2 Non-Stop Routing (NSR) router ospf <process id> [vrf <vrf name>]
nsr
Activated on a per-process basis (for both ipv4 or ipv4 VRF for PE-CE sessions)
Depends on the forwarding plane’s ability to retain state across control plane restarts and RP switchovers
Alleviates dependency on OSPFv2 protocol extensions (NSF) Neighboring routers are unaware that a router is NSR-capable
Neighboring routers are unaware that a router has gone through an RP switchover
Provides near-transparent RP switchover capability OSPF adjacencies remain up
Minimal state refreshed by the restarting router post switchover
Scalable to larger link state database sizes and number of neighbors
78
Service High Availability
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
High Availability for Advanced Service Models
Many SP Services already go beyond standard L3VPN / L2VPN / transport services
Increasing subscriber management capabilities and L4-L7 services
Examples:
Subscriber Management
Multicast
Session Border Controller
Firewall
IPSec
LI
Some Services can be made highly-available using Intra-chassis redundancy (e.g. IPSec, Firewall, NAT, PPPoX, L2TP)
Stateless inter-chassis redundancy available for BNG
Stateful Inter-chassis redundancy available for NAT, Firewall and SBC on the Cisco ASR 1000
80
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
L3VPN Key HA Technologies
Physical Circuit Diversity
Multihoming
Link Detection IP Event Dampening
BFD
Chassis Redundancy NSF/NSR
Routing Protocols and Convergence BGP PIC Core
BGP PIC Edge
IP-FRR
MPLS Path Protection
Edge Access
Mobile
CPE
Business
VRF Green
Core
Business
VRF Blue
Business
VRF Red
Business
VRF Orange
Access
Mobile
CPE
Business VRF Green
Business
VRF Blue
Business
VRF Red
Business
VRF Orange
Edge
Site A
Site B
Site C
Site D
For Your Reference
81
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
L3VPN Key HA Technologies
CPE
BFD for PE-CE Link
Detection
NSF/NSR for Chassis HA
PE Multihoming Intra-Site PE for PE Diversity
Inter-Site for SP Facility Diversity
Access
Circuit Diversity - Physical
Diversity for Multihomed CPE
Physical Circuit Diversity is Not the
Default
Must be Requested from the SP
Edge
BFD for PE-CPE / PE-P Link
Detection
NSF/NSR for Chassis HA
IP Event Dampening for PE-CPE
IP-FRR for PE-P For Cost Effective PE-P Bandwidth
BGP PIC Core
BGP PIC Edge for Multi-Homed CPE
Edge Access
Mobile
CPE
Business
VRF Green
Core
Business
VRF Blue
Business
VRF Red
Business
VRF Orange
Access
Mobile
CPE
Business VRF Green
Business
VRF Blue
Business
VRF Red
Business
VRF Orange
Edge
Site A
Site B
Site C
Site D
82
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
L2VPN — Pseudowire Redundancy
Active-Standby PW Access Circuit Redundancy
L2TPv3 and MPLS Support
Detection Mechanisms
IGP Convergence for Remote PE Failure
LDP Signaling for PE-CE Failure
LDP Timeout for Remote PE Software Failure
CE
P
P
P
CE
PE1 PE2
PE23
P
Standby PW
Active PW
BRKSPG-2207
83
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Multicast High-Availability Behavior
Before failure
Multicast state is synchronized from RPact to RPsby
Configuration
MLDv1/v2 state information
PIM or MRIB state are NOT synchronized
MFIB also synched to ESPact and ESPsby
After failure
RPsby sends out PIM hellos to all neighbors
PIM neighbors re-send PIM state
Newly active RP re-builds the PIM state
IGP reconverges to assure uRPF check
MFIB and ESP updates proceed to incorporate refreshed PIM state
ESPact continues to forward multicast traffic based on its version of the MFIB
Forwarding of multicast packets is NOT disrupted
RPact
FMRP
QFP Client
RPsby
SPAs
RF IDB State Update Msg
IOSact IOSsby
I P C
I P C
CF
Driver/Media Layer
Mcast
CEF
Config
…
MLD
ESPsby ESPact
QFP Client
FMRP
FMESP FMESP
MFIB
RF
CF
IDB State Update Msg
Driver/Media Layer
Mcast
CEF
Config
…
MLD
MFIB MRIB MFIB
MFIB
MRIB
84
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Receiver
Multicast join on primary path
Multicast join on backup path
Data packets are received from the primary and secondary paths
The redundant packets are discarded at topology merge points due to RPF checks
Failure:
Interface chance on where packets are accepted
Backup path interfaces become ‘active’
Multicast only Fast Re-Route (MoFRR)
POP1
POP2 POPN
IPTV source
Normal path
Alternate PIM
Configuration and Restrictions Dependency on ECMP and will not work without it Disabled by default and enabled through a cli Applicable to IPv4 multicast only and not IPv6 multicast Works only for SM S,G and SSM routes Works where the rpf lookups are done in a single vrf Extranet routes are not supported Both primary and secondary paths should exist in the same multicast topology.
85
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Stateful Application Switchover: PPP
Copies state information for PPP, PPPoE, and PPPoEoVLAN Sessions
Switch-over is transparent to peers Sessions are not torn-down / re-established
PPP, PPPoE, and PPPoEoVLAN Session States: Configuration (through config synch), including QoS configuration, ACLs
Session identifiers
PADR frame (cached)
RADIUS session attributes
Physical interface
VAI identifier
MD5 signature
Statistics are synchronized on ASR 1000!
86
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Stateful Application Switchover: L2TP
RPact synchronizes state with RPsby
State includes configuration, PPP session IDs, L2TP CC sequence numbers
etc.
Sequence numbers (Ns, Nr) for L2TP Control Connections (CC) are only
synched once for a packet window of X (i.e. once every X L2TP control packets)
ESP
L2TP Control Connections
RPsby
L2TP Tunnel
LNS
RPact
87
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Residential
STB
BNG Cluster
BNG Service Edge High Availability
PPP Smart Server Selection allows user to configure specific PADO delay for a received PADI packet
Can be configured per bba-group or based on circuit-id/remote-id
In case of an outage of a BNG in the cluster, other BNG stand ready to accept subscriber sessions
Detection of failure possible at both ends of PPPoE session because of missing keepalives
Subscriber sessions have to be re-established
Allows BNG redundancy with predictable behavior
E-DSLAM
Ethernet Aggregation
PADI 1
PADI
PADO
PADI PADI
PADO
PADO
2 PADO
3 PADR
4 PADS
PADR
PADS
Delay
88
Stateful Inter-Chassis
Redundancy
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Motivation for Stateful Application Inter-Chassis
Redundancy
Current Intra-chassis HA typically protects against
Control Plane (RP) Failures
Forwarding Plane (ESP) failures
Interface failures can be mitigated using link bundling (e.g. GEC)
Any other failures may result in recovery times O(hours)
Inter-chassis redundancy provides additional resilience
against
Interface Failures
System failures
Site failures (allowing for geographic redundancy)
RP
RP FP
FP
SIP
SIP
SIP
SIP
90
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Stateful System Redundancy Models
Different deployment models
1+1 – one system is actively processing and passing traffic, the other in standby mode.
1:1 – two systems are actively processing and passing traffic, and backing each other up
N+1 – N systems are actively processing and passing traffic, and share a single standby
System vs. Application
Is the inter-chassis resilience applicable to ALL of the features / functions configured on the system, or only for a particular application?
System-level: provide resilience for ALL applications and traffic configured on a system
Application-Level: provide resilience for a particular application and its traffic
Hot-standby vs. Cold-standby
Cold-standby: FIB / adjacency updates are NOT synchronized between active and standby system
Hot-standby: forwarding/state information is synchronized between active and standby system
Different Approaches can also be categorized into
Control plane active-standby / active-active
Forwarding plane active-standby / active-active
For Your Reference
91
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
System Level
Redundancy
Application Level
Redundancy Example: VSS
Failover Granularity at the System Level
Control-plane active-standby
Active RP considers ‘remote’ linecards under its
control
Forwarding-plane active-active
No application granularity for failover
Need to ensure all features are SSO capable
Example: RG Infra
Failover Granularity at the Application Level (NAT,
Firewall, SBC etc)
Control plane active-active
Each RP only considers its own linecards, but synchronizes
application state
Forwarding-plane active-active
E.g. can have one set of firewall services resilient,
and other set of firewall services non-resilient
RPact
Fabric
LC
Fabric
LC LC LC
RPsby RPact RPsby
Failover
RPact
ESP
SIP
ESP
SIP SIP SIP
RPact
Failover
FW FW
92
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
RP
Stateful INTRA-Chassis Redundancy Revisited
Building-blocks required to achieve stateful interchassis redundancy are for ASR 1006 / 1013:
1. Redundant Hardware components
RP / ESP / ESI links
SIPs/SPAs are NOT redundant
2. Forwarding / Application State Tables
3. Control mechanism to synchronize between active-standby components
Who is active / who is standby?
Initiate failover in case of failure
4. State transfer mechanism
Copy forwarding / application state tables to standby and keep synchronized
Currently provided by IOS RF/CF infrastructure over IPC
5. Failure detection mechanism
Interrupts
RP
ESP ESP
SIP SIP SIP
IPC
Active Enhanced Serdes Link ESI (internal dataplane) Standby Enhanced Serdes Link ESI (internal dataplane)
Note: EOBC (internal control plane) infrastructure NOT shown.
RT RIB NAT RT RIB NAT
93
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
nV Edge Overview Control Plane EOBC Extension (L1 or L2 connection) One or two 10G/1G from each RSP
Control plane EOBC extension is through special 1G or 10G EOBC ports on the RSP. External EOBC could be over dedicated L1 link, or over port-mode L2 connection
Data plane extension is through regular LC ports (it can even mix regular data ports and inter-chassis data plane ports on the same LC)
Doesn’t require dedicated fabric chassis flexible co-located or different location deployment, lower cost
Special external EOBC 1G/10G ports on RSP
Active
RSP
Secondary
RSP
LC LC LC LC
0 Standby
RSP
Secondary
RSP
LC LC LC LC
1
Inter-chassis data link (L1 connection) 10G or 100 G bundle (up to 32 ports)
Regular 10G or 100G data ports
Internal EOBC
For redundancy purpose, minimal two control plane and two data plane links are required
External EOBC link fail won’t cause RP failover as long as it has alternative EOBC link
94
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Inter-Chassis Control Plane and Data Plane
Packet Format Inter-chassis control plane link
Ethernet snap with special ethertype and internal mac addresses
Work over L2 circuit as well assuming it’s port mode: transparently forward every packet
Recommend L1 link, with up to 10msec latency
Inter-chassis data plane link
Regular Ethernet frame, with 802.1q tag (VLAN=1)
In theory, it can work over L2 circuit, but it’s never tested and won’t be supported officially
95
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Multi-chassis LAG
mLACP uses ICCP to synchronize LACP configuration & operational state between PoAs, to provide DHD the perception of being connected to a single switch
All PoAs use the same System MAC Address & System Priority when communicating with DHD
Configurable or automatically synchronized via ICCP
Every PoA in the RG is configured with a unique Node ID (value 0 to 7). Node ID + 8 forms the most significant nibble of the Port Number
For a given bundle, all links on the same PoA must have the same Port Priority
ICCP
DHD
PoA1
LACP PoA2
Node ID: 1
Node ID: 2
Port #: 0x9001, Port Priority 1
Port #:0xA001, Port Priority 2
System MAC: aaaa.bbbb.cccc
System Priority: 1
96
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Inter-chassis Communication Protocol
ICCP allows two or more devices to form a ‘Redundancy Group’
ICCP provides a control channel for synchronizing state between devices
ICCP uses TCP/IP as the underlying transport ICCP rides on targeted LDP session, but MPLS need not be enabled
Various redundancy applications can use ICCP: mLACP
Pseudowire redundancy
RG
ICCP over Dedicated Link
RG
ICCP over Shared Network
97
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Control Plane HA Model
Only one Active RSP, Only one standby RSP at a given time, which are located on two different chassis
SSO/NSF/NSR works exactly the same way as two RSPs on the same chassis
Reliable out of band control channel between two chassis
IOS-XR control plan can tolerant hundreds of msec latency*, although the latency can impact overall service convergence time
Virtual Chassis is always on as long as there is one chassis and one RSP alive
Active
RSP
Secondary
RSP
LC LC LC LC
0 Standby
RSP
Secondary
RSP
LC LC LC LC
1
DSC Chassis Non DSC Chassis
Active control plane
Standby control plane
Standby
RSP
Active
RSP
Standby
RSP
* Practically, recommend maximum 10msec latency between two chassis
For Your Reference
98
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Data Plane Forwarding Model
Inter-chassis data links simulate the switch fabric , which provide the data connection between two chassis. It has similar features as switch fabric, for example, fabric qos. Packet load balancing over inter-chassis links is same as regular link bundle: per-flow based
Keep the existing IOS-XR two-stage forwarding model no forwarding architecture change for single chassis vs. nV Edge system
In case of ECMP or link bundle paths cross two chassis, it prefer local port instead of load balancing packet to the other chassis. This is to reduce the inter-chassis link usage as much as possible. However, this feature (local rack preference) could be turn off by user CLI
Only single Multicast copy is sent over inter-chassis link. Multicast replication is done on egress line cards and fabric on the local chassis
Active
RSP
Secondary
RSP
LC LC LC LC
0 Standby
RSP
Secondary
RSP
LC LC LC LC
1
Simulated switch fabric
For Your Reference
99
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Data Forwarding
Ingress LC
P2
P1
Data Plane
Lookup
1
Lo
ad
B
ala
nce
Inter-Chassis LC
P2
P1
Data Plane
Encapsula
tion
Egress LC
Data Plane
LO
OK
UP
Inter-Chassis LC
P2
P1
Data Plane E
ncapsula
tion
Inter-Chassis LC
Data Plane
De
ca
psu
lat
ion
P2
P1
Inter-Chassis LC
Data Plane
De
ca
psu
lat
ion
P2
P1
P2
P1
2
3
3
4
4
5
Inte
r-C
has
sis
Lin
k b
un
dle
Chassis 0 Chassis 1
Ingress Forwarding Lookup L2/L3/Mcast regular lookup
Inter-Chassis Load Balance Load balance across multiple inter-chassis links
Inter-Chassis Encapsulation Egress Forwarding Lookup L2/L3/Mcast regular lookup
1
2
3
4
5
Inter-Chassis Decapsulation
For Your Reference
100
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Introduction to RG-Infra
RG Infra is the IOS Redundancy Group Infrastructure to enable the synchronization of application state data between different physical systems
Does the job of RF/CF between chassis
Infrastructure provides the functions to
Pair two instances of RG configured on different chassis for application redundancy purposes
Determine active/standby state of each RG instance
Exchange application state data (e.g. for NAT/Firewall)
Detect failures in the local system
Initiate & manage failover (based on RG priorities, allows for pre-emption)
Assumptions
Application state has to be supported by RG infra (ASR 1000 currently supports NAT, Firewall, SBC)
Connectivity redundancy solved at the architectural level (need to ‘externalize’ the redundant ESI links of
the intra-chassis redundancy solution)
101
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Redundancy Groups Functions
Registers applications as clients
Registers (sub)interfaces / {SA/DA}-tuplets in case of firewall
Determines if traffic needs to be processed or not E.g. for Firewall: if a subset of sessions are associated with a RG in active state, then the Firewall application will perform normal
processing for those sessions and actively sync the session state to another device that has the same RG in STANDBY state.
For Firewall sessions that are associated with a RG in STANDBY state, the session information will be synchronized from a device
that has the RG in ACTIVE state.
Communicates control information between RGs using a redundancy group protocol Advertisement of RGs and RG state
Determination of peer IP address
Determination of presence of active RG
Synchronizes application state data using a transport protocol
Manages Failovers! RPact
ESP
SIP
FW RG
SPA SPA
SIP SPA SPA
RPact
ESP
SIP
RG
SPA SPA
SIP SPA SPA
FW
RG state data
RG control data
Active
102
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Redundancy Groups Functions — Details
Configuration of stateful system redundancy Priority (similar to HSRP priority for RG state determination)
Preemption, Name
RG State control Init, Active, Standby, disabled
Communicating state changes to other software entities in the system (e.g. QFP software)
Synchronization management Synchronization state tracking (standby has to request bulk-updates from active)
Determines when synchronization is started (e.g. ensures transport is available)
Peer Management Maintain information about peers
Fault Handling Changing priorities of RG (may affect RG state)
Fault event dampening
Logging
Integration with Enhanced Object tracking / BFD
Transport Connectivity Knows via which interface application state is synchronized
Can be different for application state data and RG control messages
For Your Reference
103
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Intra-Chassis vs. Inter-Chassis Redundancy
Function / Method Stateful Intra-chassis Stateful Inter-chassis
Hardware redundancy ESP, RP ESP, RP, Interfaces
Redundant connectivity Internal ESI links Redundant links to neighbor nodes
Redundancy control RF/CF RG
State synchronization IPC over EOBC External GEC
Failure detection mechanism Interrupts BFD, Hellos
Failover mechanism Chassis Manager RG Protocol (HSRP like)
For Your Reference
104
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Possible RG-Infra Redundancy Models
Active-Standby
All application traffic associated with a SINGLE RG instance
Failures would switch all traffic over to the standby chassis
Active-Active
Multiple RG instances configured per system
Subset of traffic associated with a particular RG instance
Single failure only affects subset of overall application traffic
2+1 Active-Standby
2 or more chassis loadshare application traffic, backed up by a single standby system
Subset of traffic associated with a particular RG instance on different chassis
Single failure only affects subset of overall application traffic
RGact RGsby
RG1act RG1
sby
RG2act RG2
sby
RG3act
RG3sby
RG1act
RG1sby
RG2act RG2
sby
105
Case Studies
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Case Study: Highly Available IP Architecture for
Mobile
Cellsite
Agg1 CSR
MSPP MSPP
MME
SGW
MSC
RNC
LTE Core
CDMA Core
PE
PE
EvDO/LTE VRF
1xRTT VRF
QFP
MTSO
MPLS VPN
VRF 1xRTT
VRF EvDO/LTE
Agg2
Service Termination
GE 10 GE QFP
Internet Core
L2 Domain L3 Domain L3 Domain
Local VLANs or T1s
Transport VLANs / Static Routes / BGP PIC Edge / BFD protection OSPF/RIP/VRRP
FE T1
EoMPLS Backhaul
One Second Convergence Requirement
CSR EvDO/LTE VRF
1xRTT VRF
QFP
107
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Agg1 CSR
MSPP MSPP
MME
SGW
MSC
RNC
LTE Core
CDMA Core
PE
PE
EvDO/LTE VRF
1xRTT VRF
QFP
MPLS VPN
VRF 1xRTT
VRF EvDO/LTE
Agg2
Internet Core
Static Routes establish connectivity between loopback addresses
VRF VRF
VRF
VRF
Static routes for cellsite reachability BGP PIC Edge for Layer-3 convergence VRRP for MTSO
VRF Tables Updated
BFD sessions established
CSR EvDO/LTE VRF
1xRTT VRF
QFP
Case Study: Highly Available IP Architecture for
Mobile — Transport
108
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Agg1 CSR
MSPP MSPP
MME
SGW
MSC
RNC
LTE Core
CDMA Core
PE
PE
EvDO/LTE VRF
1xRTT VRF
QFP
MPLS VPN
VRF 1xRTT
VRF EvDO/LTE
Agg2
Internet Core
CSR EvDO/LTE VRF
1xRTT VRF
QFP
Steady state: CSR distributes flows across both Agg’s using ECMP. Traffic could flow across Agg inter switch links. Each Agg handles traffic related to all services from the cell-site.
1xRTT
1xRTT
EVDO
EVDO
1xRTT
Case Study: Highly Available IP Architecture for
Mobile — Steady-State Traffic Flows
109
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Case Study: Highly Available IP Architecture for
Mobile — Link Failure Agg1 CSR
MSPP MSPP
MME
SGW
MSC
RNC
LTE Core
CDMA Core
PE
PE
EvDO/LTE VRF
1xRTT VRF
QFP
MPLS VPN
VRF 1xRTT
VRF EvDO/LTE
Agg2
Internet Core
CSR EvDO/LTE VRF
1xRTT VRF
QFP
Steady state: Traffic flows distributed across both Agg. Failure: GE link from MSPP to Agg1 fails. Action:
BFD session to Agg1 times out at CSR. Agg1 next hop removed from forwarding table. Traffic flows resume across existing path to Agg2.
Results: Traffic flows to Agg1 via Agg2.
No changes to LAN side connectivity
x
110
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Case Study: Highly Available IP Architecture for
Mobile — Aggregation Switch Failure Agg1 CSR
MSPP MSPP
MME
SGW
MSC
RNC
LTE Core
CDMA Core
PE
PE
EvDO/LTE VRF
1xRTT VRF
QFP
MPLS VPN
VRF 1xRTT
VRF EvDO/LTE
Agg2
Internet Core
CSR EvDO/LTE VRF
1xRTT VRF
QFP
x
Steady state: Traffic flows distributed across Agg. Failure: Agg1 power outage. Action:
BFD and VRRP sessions time out BGP and OSPF neighbors drop due to BFD BGP PIC Edge ensures sub-second convergence Traffic flows resume across existing path thru Agg2.
Results: Traffic flows via Agg2 to end hosts. 111
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Case Study: Highly Available IP Architecture for
Mobile — CSR Failure Agg1 CSR
MSPP MSPP
MME
SGW
MSC
RNC
LTE Core
CDMA Core
PE
PE
EvDO/LTE VRF
1xRTT VRF
QFP
MPLS VPN
VRF 1xRTT
VRF EvDO/LTE
Agg2
Internet Core
CSR EvDO/LTE VRF
1xRTT VRF
QFP
x
Steady state: Traffic flows distributed across CSR. Failure: CSR power outage. Action:
BFD sessions time out BGP neighbors drop due to BFD Mobile handsets resync to neighboring cell site
Results: Mobile handset voice connectivity is maintained.
112
Summary
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Summary
Motivation for High Availability in SP Aggregation Networks
Network Level High Availability
System High Availability
Service High Availability
Stateful Inter-chassis Redundancy
Case Studies
Summary and Conclusions
114
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Key Takeaways
High-Availability becoming increasingly deployed in Aggregation Networks
Motivated by experiences with MPLS Core Networks
Many high-availability techniques deployed in the core are now applied in MPLS aggregation networks
MPLS TE FRR, BFD, EOAM, Pseudowire Redundancy …
Service High Availability requires comprehensive approach including the deployment of
Network level resiliency
System Level resiliency
L4-7 service resiliency
Stateful Inter-chassis redundancy increasingly being considered to provide geographic redundancy for applications
High Availability comes at a cost (CAPEX & OPEX)!
115
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Call to Action
• Visit the Cisco Campus at the World of Solutions to experience Cisco innovations in action
• Get hands-on experience attending one of the Walk-in Labs
• Schedule face to face meeting with one of Cisco’s engineers
at the Meet the Engineer center
• Discuss your project’s challenges at the Technical Solutions Clinics
116
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public 117
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Recommended Reading
N. Stringfield et. Al, “Cisco Express Forwarding”, ISBN-13: 978-1-58705-236-1
D. C. Lee, “Enhanced IP Services for Cisco Networks”, ISBN-13: 978-1-57870-106-3
A. Sayeed, M. Morrow, “MPLS and Next-Generation Networks”, ISBN-13: 978-1-58720-120-2
J. Davidson et. al, “Voice over IP Fundamentals”, 2nd Edition, ISBN-13: 978-1-58705-257-6
V. Bollapragada et. Al, “Inside Cisco IOS Software Architecture “, ISBN-13: 978-1-57870-181-0.
R. Wood, “Next-generation Network Services”, ISBN-13: 978-1-58705-159-3.
K. Lee, F. Lim, B. Ong, “Building Resilient IP Networks”, ISBN-13: 978-1-58705-215-6
T. Szigeti, C. Hattingh, “End-to-End QoS Network Design: Quality of Service in LANs, WANs, and VPNs:, ISBN-13: 978-1-58705-176-0
B. J. Carroll, “Cisco Access Control Security”, ISBN-13: 978-1-58705-124-1.
A. Khan, “Building Service-Aware Networks: The Next-Generation WAN/MAN”, ISBN-13: 978-1-58705-788-5
118
© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public
Whitepapers on CCO
Cisco IOS High Availability
http://www.cisco.com/en/US/tech/tk869/tk769/tech_white_papers_list.html
http://www.cisco.com/en/US/products/ps6550/prod_white_papers_list.html
Campus Network for High Availability Design Guide
http://www.cisco.com/en/US/docs/solutions/Enterprise/Campus/HA_campus_DG/hacampusdg.html
Cisco Validated Designs
http://www.cisco.com/en/US/netsol/ns742/networking_solutions_program_category_home.html
ASR 9000
Cisco ASR 9000 Series High Availability: Continuous Network Operations
Introduction to Cisco ASR 9000 Series Network Virtualization Technology
Distributed Virtual Data Center for Enterprise and Service Provider Cloud
ASR 1000
Cisco ASR 1000 Series Aggregation Services Routers
Cisco ASR 1000 Series: ISSU Deployment Guide and Case Study
Cisco Unified Border Element (SP Edition) on Cisco ASR 1000 Series
Cisco Unified WAN Services: Services, Security, Resiliency, and Intelligence
119