© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Unified MPLS: Advanced Scaling for Core and Edge Networks BRKSPG-2405
1
Rajiv Asati Distinguished Engineer
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Abstract
Service Providers (SPs) are striving towards becoming 'Experience Providers' while offering many residential and/or commercial services. Many SPs have to build an agile Next Gen Networks (NGN) that can optimally deliver the 'Any Play' promise. However, as the Networks continue to get are getting bigger, fatter and richer, some of the conventional wisdom of designing IP/MPLS networks is no longer sufficient. This session introduces a 'Cisco Validated Design' for building Next-Gen Networks' Core and Edge. It briefly discusses the technologies integral to such a design and focus on their implementation using IOS-XR platforms (CRS-1/3 and ASR 9000). The session looks at the scaling designs and properties of IP, MPLS, the IGP and BGP as well as the protection mechanisms IP/LDP FRR and MPLS-TE FRR.
This session is intended to cover - - Unicast routing + MPLS design - Fast Restoration - Topology Dependency - Test Results - Case Study
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Agenda
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Dependency
‒ Test Results
Case Study
Conclusion
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 4
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Dependency
‒ Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Introduction Trend Networks becoming larger
‒ Quad-play (Video, Voice, Data & Mobility) ‒ Merger & Acquisition ‒ Growth
Exponential bandwidth consumption ‒ Business Services ‒ Mobile
MPLS in the Access ‒ Seamless MPLS ‒ MPLS-TP
BGP ASN consolidation ‒ Single ASN offering to customers
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Introduction NGN Requirements Large Network
‒ 2000+ routers, say
Multi-Play Services Anywhere in network ‒ Service Instantiation happens anywhere
End-to-End Visibility ‒ v4/v6 Uni/Multicast based Services
Fast Convergence or Restoration ‒ Closer to Zero loss, the better.
Scale & Performance
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 7
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Dependency
‒ Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Solution Overview
Unicast Routing + MPLS - Divide & Conquer 1. Isolate IGP domains
2. Connect IGP domains using BGP
Fast Restoration – Leverage FRR 1. IP FRR (IGP LFA & BGP PIC)
2. MPLS FRR (LDP FRR & TE FRR)
Topological Consideration – Choose it right 1. PoP Design
2. ECMP vs. Link-Bundling
Services – Scale
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 9
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Dependency
‒ Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Must Provide…. PE-to-PE Routes (and Label Switched Paths)
‒ PE needs /32 routes to other PEs
‒ PE placement shouldn’t matter
Single BGP ASN
Backbone
Aggregation
.
Access Region 2
.
PE31
R
PE21
Access
.
Region1
.
Aggregation
PE11
PE21
LSP
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Conventional Wisdom Says… Advertise infrastructure (e.g. PE) routes in IGP
Advertise infrastructure (e.g. PE) labels in LDP
Segment IGP domains (i.e. ISIS L1/L2 or OSPF Areas)
Aggregation
.
Access Region 2
.
PE31
R
PE21
Access .
Region1
.
Aggregation
PE11
PE21
Backbone
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
BGP for Services End-to-End IGP & LDP for Infrastructure
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Conventional Wisdom Not Good Enough… Large IGP database size a concern
‒ For fast(er) convergence
Large IGP domain a concern ‒ For Network Stability.
Large LDP database a concern
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design ‘Divide & Conquer’ – Game Plan
Disconnect & Isolate IGP domains ‒ No more end-to-end IGP view
Leverage BGP for infrastructure (i.e. PE) routes ‒ Also for infrastructure (i.e. PE) labels
Backbone Aggregation
.
Access Region 2
.
PE31
R
PE21
Access .
Region1
.
Aggregation
PE11
PE21
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
Isolated IGP & LDP Isolated IGP & LDP Isolated IGP & LDP BGP for Infrastructure
BGP for Services
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer – End Result
Example - ‘PE31’ Reachability
Control Plane Flow – RIB/FIB Table View
Data Plane Flow – PE11 to PE31 Traffic View
Backbone Aggregation
.
Access Region 2
.
R
PE21
Access .
Region1
.
Aggregation
PE21
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
PE31 :: Next-Hop = P1; BGP; Label = L100; BGP P1 :: Next-Hop = P11; IGP Label = L200; LDP
PE31 :: Next-Hop = P2; BGP Label = L101; BGP P2:: Next-Hop = P100; IGP Label = L201; LDP
PE11
P1
P11
P2
PE31 :: Next-Hop = P31; IGP Label = L110; LDP
PE31
IP L200 L100
IP L110
IP IP L201 L101
P100
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer – Summary 1. IGP is restricted to carry only internal routes
‒ Non-zero or L1 area carries only routes for that area
‒ Backbone carries only backbone routes *
2. PE redistributes its loopback into IGP as well as iBGP+Label 3. PE peers with its local ABRs using iBGP
‒ ABRs act as Route-reflectors
‒ ABRs reflect _only_ Infrastructure (i.e. PE) routes
4. ABR, as RR, changes the BGP Next-hop to itself ‒ On every BGP advertised routes
5. PEs separately peer for Services (VPN, say)
* ISIS L1->L2 (or L1->L1) Redistribution Cannot Be Avoided Yet, but OSPF Non-Zero<->Zero Area Redistribution Can Be.
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer
1. IGP is restricted to carry only the internal routes ‒ Non-zero or L1 area carries only routes for that area
‒ Backbone carries only backbone routes *
Backbone Aggregation
.
Access Region 2
.
PE31
R
PE21
Access .
Region1
.
Aggregation
PE11
PE21
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
1
* Unlike OSPF, ISIS Backbone Would Carry Both L1 and L2 Routes Since L1->L2 (or L1->L1) Redistribution Cannot Be Avoided Yet
ABR ABR
Isolated IGP Isolated IGP Isolated IGP
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer
1. PE redistributes its loopback into IGP as well as iBGP+Label
Backbone Aggregation
.
Access Region 2
.
PE31
R
PE21
Access .
Region1
.
Aggregation
PE11
PE21
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
2
Loopback Int Redistributed into IGP and BGP+Label
ABR ABR
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer
1. PE peers with its local ABRs using iBGP+label ‒ ABRs act as Route-reflectors
‒ ABRs reflect _only_ Infrastructure (i.e. PE) routes
‒ RRs also in the backbone
Backbone Aggregation
.
Access Region 2
.
PE31
R
PE21
Access .
Region1
.
Aggregation
PE11
PE21
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
iBGP+Label Peering
3
ABR ABR
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer
1. ABR, as RR, changes the BGP Next-hop to itself ‒ On each BGP advertised routes
Backbone Aggregation
.
Access Region 2
.
PE31
R
PE21
Access .
Region1
.
Aggregation
PE11
PE21
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
ABR Sets BGP NH to Itself ABR Sets BGP NH to Itself
4
ABR ABR
BGP Prefix PE31: Next-Hop = P1; Label=L100
BGP Prefix PE31: Next-Hop = P2; Label=L101
BGP Prefix PE31: Next-Hop = PE31; Label=Null
P1 P2
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer
1. PEs separately peer using iBGP for Services ‒ Dedicated RRs for IPv4/6, VPNv4/6, L2VPN, etc.
‒ More Details on BGP Scale for Services Later…
Backbone Aggregation
.
Access Region 2
.
PE31
R
PE21
Access .
Region1
.
Aggregation
PE11
PE21
ISIS Level 2 Or
OSPF Area 0 . .
5
ABR ABR
RRs RRs RRs
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Routing + MPLS Design Divide & Conquer – End Result
Example - ‘L3VPN Services’
PE11 sends L3VPN traffic for an L3VPN prefix “A” to PE31
Backbone Aggregation
.
Access Region 2
.
R P31
Access .
Region1
.
Aggregation
PE21
ISIS Level 2 Or
OSPF Area 0
ISIS Level 1 Or
OSPF Area X
ISIS Level 1 Or
OSPF Area Y
PE31 :: Next-Hop = P2; BGP Label = L101; BGP P2:: Next-Hop = P100; IGP Label = L201; LDP
PE11
P11
P2
PE31 :: Next-Hop = P31; IGP Label = L110; LDP
PE31
IP L200 L100
IP L30
IP IP L101 L30
P100
L30
P1
L3VPN “A”:: next-Hop = CE31; IGP Label = Unlabel
IP
IP L100 L30
L201 L110
L30
PE31 :: next-hop = P1; BGP; label = L100; BGP P1 :: Next-hop = P11; IGP label = L200; LDP
L3VPN “A” Next-Hop = PE31; BGP Label = L30 ; BGP PE31 :: Next-Hop = P1; BGP; Label = L100; BGP P1 :: Next-Hop = P11; IGP Label = L200; LDP
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 22
Routing + MPLS Design Take-Away
Higher Network scale is attainable ‒ 1000s of routers
BGP and MPLS Label Stacking are key
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 23
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Dependency
‒ Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration
Business Services demanding faster restoration ‒ Against link or node failures
“Service Differentiator” for many operators
Faster Restoration is driving towards 0 loss ‒ ~50ms restoration may be good enough for many
‒ Requirements influence Complexity and Cost
Fast Restoration is optimal with “Local Protection” ‒ pre-compute and pre-install alternate path
‒ no need for remote nodes to know about the failure
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration
Fast Restoration of Services i.e. BGP Prefixes ‒ BGP Prefix Independent Convergence (PIC)
Fast Restoration of BGP next-hops i.e. IGP Prefixes ‒ IP FRR (LFA) with LDP FRR (or RSVP-TE FRR)
Fast Convergence (FC) of IP routing protocols is key and still required
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration vs. Fast Convergence
Detection (link or node aliveness, routing updates
received) State propagation
(routing updates send)
Walkthrough routing
DB’s Compute primary path & label Download
to HW FIB
Switch to newer path
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration vs. Fast Convergence
Detection (link or node aliveness, routing updates
received) State propagation
(routing updates send)
Walkthrough routing
DB’s Compute primary path & label Download
to HW FIB
Switch to newer path
Switch to Repair Path
Pre-Compute Repair path Download
to HW FIB
Offline Calculation
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Remember that FRR is intended for temporary restoration Fast Convergence (FC) is key for IP routing protocols Faster the routing convergence, faster the permanent
restoration ‒ <1sec restoration is possible
Routing convergence happens at the process level, hence, depends on the platform processor ‒ Restoration time can not be guaranteed
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
Fast Convergence IGP Prefixes
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Detect Link/node down event as fast as possible ‒ BFD, Layer2 protocol keep-alives, Alarms, IGP fast hellos, Proactive
Protection
Generate the link state event—LSP/LSA generation is optimized Propagate the changes in the network as soon as possible—
Flooding and passing is optimized Recalculate the paths (run SPF) as soon as possible—Support
of incremental SPF and optimized for full SPF Install the new routes in the routing/forwarding table with Prefix
Prioritization CRITICAL: IPTV SSM sources
HIGH: Most Important PE’s
MEDIUM: All other PE’s
LOW: All other prefixes
Fast Convergence IGP Prefixes
MUST for
FRR
MUST for FC
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Convergence IGP Tuning for FC
OSPF Event Propagation
‒ timers pacing flood value
‒ timers pacing retransmission value
‒ default values are 33 msec/66 msec
OSPF Subsecond Hellos Configuration:
‒ ipospf dead-interval minimal hello-multiplier value
‒ Value—range 3–20
OSPF LSA Generation Exponential Backoff
timers throttle lsa all lsa-start lsa-hold lsa-max
‒ timers lsa arrival timer
OSPF SPF ExponentialBackoff
‒ Timers throttle spfspf-start spf-hold spf-max
‒ All LSA/SPF values are in ms
• IS-IS hello interval/ Hello Multiplier
isis hello-interval { seconds | minimal }
isis hello-multiplier value ------- Value—range 3–20
• IS-IS LSP-Generation Exponential Backoff
lsp-gen-interval lsp-max lsp-start lsp-hold
lsp-max—(sec) lsp-hold—(msec) lsp-start—(msec)
• IS-IS Event Propagation lsp-interval value
Default rate - one LSP every 33 ms
• Fast LSP Flooding fast-flood lsp-number (Previously ip fast-
convergence)
• IS-IS SPF Exponential Backoff spf-interval spf-max spf-start spf-hold
<spf-max>- (sec) <spf-start> - (msec) <spf-hold> - (msec)
prc-interval prc-max prc-start prc-hold <prc-max>- (sec) <prc-start> - (msec) <prc-hold> - (msec)
OSPF Tuning IS-IS Tuning
Note: MinLSArrival Must Be <= lsa-Hold
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
Reference
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration IGP Prefixes MPLS FRR and IP FRR are viable options
‒ Both pre-compute and pre-install alternate path
IP FRR (LFA) is simpler than RSVP-TE based MPLS FRR ‒ Easy to configure and manage ‒ Does not require network-wide support ‒ Has topological dependencies
IP FRR (LFA), with LDP LSP, provides simpler MPLS FRR ‒ Easy to configure and manage ‒ Does not require network-wide support ‒ Removes most of topological dependencies
Use IP FRR & LDP FRR (RSVP-TE FRR only if one have to)
‒ RSVP-TE for bandwidth engineering as usual
FRR – Fast Reroute LFA – Loop Free Alternates LSP – Label Switched Path
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IP/LDP FRR: Apply it as an intra PoP and inter PoP FRR solution
RSVP-TE FRR: Apply it as an inter PoP FRR solution, if IP/LDP FRR doesn’t give enough coverage
PoP
PoP
PoP
PoP
PE
P
P
PoP
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC Fast Restoration IGP Prefixes
Intra PoP Inter PoP
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration IGP – IP FRR
IP FRR (Loop Free Alternates) provides a pre-computed backup (aka repair path) per destination prefix
IP FRR (LFA) can be deployed in two ways : ‒ Per-Link LFA – Protects all the destinations reachable via the
protected link
‒ Per-Prefix LFA – Protects a destination against the next-hop link or node failure
IP FRR (LFA) well applies to most SP topologies ‒ http://tools.ietf.org/html/draft-ietf-rtgwg-lfa-applicability-00
Note: SPF calculations for LFAs are performed in background and pre-empted in case of any convergence events
Reference
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
A backup path for all prefixes reachable via next-hop node (F) over the protected link (S-F) ‒ 1 SPF per protected link
No node protection possible
Sub-optimal forwarding during FRR
Fast Restoration – IGP IP FRR : Per-Link LFA
S F
D
Primary link Backup link
Route D Primary Next Hop: F Backup Next Hop: R1
R1
Protecting Node Next-hop Node
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
S F
R1
D
Route D NH: F, LFA: R1
Route D NH:F
R2
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
R3
Availability of the backup NH is dependent on the topology and link metric assignments
All depends on metric assignment
10
10
10
10
10 10
Route D NH: F
LFA: no
Route D NH: S
RP/0/0/CPU0:ospf-3-2(config)#router ospf 1 RP/0/0/CPU0:ospf-3-2(config-ospf)#area 0 RP/0/0/CPU0:ospf-3-2(config-ospf-ar)#int pos 0/3/0/0 RP/0/0/CPU0:ospf-3-2(config-ospf-ar-if)#fast-reroute per-link enable
Fast Restoration – IGP IP FRR : Per-Link LFA
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
A backup path for a prefix (e.g. D) reachable via next-hop node (F) ‒ 1 SPF per neighbor
No node protection possible
Sub-optimal forwarding during FRR
Fast Restoration – IGP IP FRR : Per-Prefix LFA
S F
D
Backup path1 (link protection) Backup path2 (node protection)
Route D Primary Next Hop: F Backup Next Hop: R1
R1
Protecting Node Next-hop Node
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
By default, LFA computation is disabled
LFA needs to be enabled only on protecting router
Fast Restoration – IGP IP FRR : Per-Prefix LFA (Configuration)
! router isis fast-reroute per-prefix {level-1 | level-2} {all | route-map <route-map-name>} ! router ospf 1 fast-reroute per-prefix enable prefix-priority low !
S
router isis <instance-id> interface <type> <instance> address-family ipv4 [unicast] fast-reroute per-prefix level <1|2>
IOS
IOS-XR
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
10.0.0.0/8
20.0.0.0/8
Fast Restoration – IGP IP FRR : Per-Prefix LFA
IGP pre-computes a backup path per IGP prefix FIB pre-installs the backup path in dataplane
2
6 5
5 1
1
2
4
6
10.0.0.0/8, NH = D, cost= 10 20.0.0.0/8, NH = D, cost= 7
A
F
B
D E
C
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
10.0.0.0/8
20.0.0.0/8
Fast Restoration – IGP IP FRR : Per-Prefix LFA
10.0.0.0/8, NH = D, cost= 10 20.0.0.0/8, NH = D, cost= 7
10.0.0.0/8, NH = C, cost=11 20.0.0.0/8, NH = A, cost=9
10.0.0.0/8, NH = A, cost=14 20.0.0.0/8, NH = direct, cost=6
• IGP pre-computes a backup path per IGP prefix • FIB pre-installs the backup path in dataplane
A
F
B
D E
C
2
6 5
5 1
1
2
4
6
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
10.0.0.0/8
20.0.0.0/8
Fast Restoration – IGP IP FRR : Per-Prefix LFA
10.0.0.0/8, NH = D, cost= 10 20.0.0.0/8, NH = D, cost= 7 10.0.0.0/8, NH = D, cost=10 –
LFA: B 20.0.0.0/8, NH = D, cost=7 – LFA: F
• IGP pre-computes a backup path per IGP prefix • FIB pre-installs the backup path in dataplane
A
F
B
D E
C
2
6 5
5 1
1
2
4
6
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration – IGP LFA with LDP
The link between A and B failed.
A sends packets to C instead by swapping labelA with labelC distributed by C.
LDP requirement: Downstream Unsolicited; Liberal Retention
The backup path for destination P/p must contain the label bound by the backup neighbor
This is why, whether the IGP computes per-prefix or per-link, the RIB and FIB representation is always per-prefix ‒ this allows to store the per-path dependent backup label
Protecting Node
Link Failure
A
C
B
Primary Path Repair Path
P/p packet labelA
packet labelB
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
R2
R4
R6 R7
R5
R3
R1
Access Region
Backbone
Fast Restoration – IGP Remote LFA (aka PQ) Any node which meets the P
and Q properties
‒ P: the set of nodes reachable from R2 without traversing [R2-R4]
‒ Q: the set of nodes which can reach R4 without traversing [R2-R4]
Best PQ node
‒ The closest from R2: R5
Establish a directed LDP session with the selected PQ node
Backbone
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration – IGP Remote LFA (aka PQ) R2’s LIB
‒ R4’s label for FEC R6 = 408
‒ R1’s label for FEC R5 = 103
‒ R5’s label for FEC R6 = 502
R2’s FIB for destination R6 ‒ Primary: out-label = 408, oif = R4
‒ Backup: out-label = 502
oif = [push 103, oif = R1] R2
R4
R6 R7
R5
R3
R1
Access Region
Backbone
103
408
502
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration RSVP-TE FRR
RSVP-TE FRR link protection (and prefix independent): <50ms
Easy to operate with auto-tunnel
RSVP-TE FRR node protection (and prefix independent):
<100ms (depends on time to detect the node failure)
RSVP-TE FRR path protection (and prefix independent):
Time depends of time to signal the path error to the head end (not a local mechanism)
Challenging to operate (due to due to its end-end / 1:1 protection)
Appropriated to specific scenario
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
Note: RSVP-TE Provides FRR Mechanism as well as: • Bandwidth Management • Traffic Engineering • Is not Topology Dependent like IP LFA
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Fast Restoration RSVP-TE FRR – Link Protection
Router C
Router D Router A Router B Router E
interface Tunnel0 tunnel destination Router D … explicit-path R2-R3-R4 notunnel mpls traffic-eng autoroute announce
interface POS0/0 mpls traffic-eng backup-path Tunnel0
interface Tunnel0 tunnel destination Router E .. etc ... tunnel mpls traffic-eng fast-reroute
x
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
What if Router D failed?
Link protection would not help as the backup tunnel terminates on Router D (which is the NHop of the protected link)
Protect tunnel to the next hop PAST the protected link (NNhop)
Router D
Router C
Router A Router B Router E
Fast ReRoute Backup Tunnel
NHop
Protected Link
Router F
Fast ReRoute Backup Tunnel
NNHop
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC Fast Restoration RSVP-TE FRR – Node Protection
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
IP FRR RSVP-TE/MPLS FRR
1 Repair Path Least Cost
Constraints Based with Bandwidth Guarantee and Path Control
2 SRLG Capable Capable 3 Link Protection Capable Capable 4 Node Protection Capable Capable 5 Path Protection Not Capable Capable 6 Control Plane Requirement None with LFA RSVP-TE
7 Load Distribution over Multiple Repair Paths
Capable Not Capable
8 Provisioning Complexity Minimal, If Any Significant 9 Topology Dependency Yes No
Fast Restoration IP FRR (LFA) vs. RSVP-TE FRR
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 54
Fast Restoration BGP PIC (Prefix Independent Convergence)
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
What Is PIC or BGP FRR?
PIC provides a fast convergence functionality upon failure to cutover to any backup path within sub-seconds independent of the number of prefixes
BGP Fast Reroute (BGP FRR)—enables BGP to use alternate paths within sub-seconds after a failure of the primary or active paths
PIC or FRR dependent routing protocols (e.g. BGP) install backup paths
Without backup paths
‒ Convergence is driven from the routing protocols updating the RIB and FIB one prefix at a time - Convergence times directly proportional to the number of affected prefixes
With backup paths
‒ Paths in RIB/FIB available for immediate use
‒ Predictable and constant convergence time independent of number of prefixes
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Site2
Site1
1
3
1. PIC Core – When IGP Path Changes 2. PIC Edge – When Remote PE Node (or Its Reachability)
Fails 3. PIC Edge – When PE-CE Link Fails
PIC Edge vs. PIC Core
PE3
PE1
PE2 2
CE1 CE2
PIC Core CLI on 7600 - cef table output-chain build favor convergence-speed
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
BGP PIC Edge PE-CE Link Protection
PE1 and PE2 pre-compute bgp backup paths using bgp best-external approach
When primary link PE1 - CE1 fails: ‒PE1 holds on to the bgp local labels and re-routes CE1’s traffic to PE2 using labels advertised by PE2
‒ Uses fixed timer to clean up stale local labels
‒ PE3 is expected to converge and start using PE2’s label to send traffic to CE1
CE2
PE1
PE2
CE1 PE3
MPLS-VPN
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
Normal Path Backup Path
router bgp 100 address-family ipv4 vrf V1 bgpadvertise-best-external
router bgp 100 address-family ipv4 vrf V1 bgp additional-paths install
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
PE1, PE2 and PE3 precomputes bgp backup
When node PE1 fails: ‒ IGP notification on PE3 invalidates active path
‒ Switches to backup path
‒ PE3 is expected to converge to start using PE2’s label to send traffic toCE1
CE2
PE1
PE2
CE1 PE3
MPLS-VPN
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
Normal Path Backup Path
BGP PIC Edge PE-Node Protection
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Convergence With and Without PIC BGP PIC Core and PIC Edge
Without PIC : Convergence is a function of number of affected prefixes during failure
With PIC : Convergence is predictable and remains constant independent of the number of prefixes
Core
1
10
100
1000
10000
100000
1
2500
0
5000
0
7500
0
1000
00
1250
00
1500
00
1750
00
2000
00
2250
00
2500
00
2750
00
3000
00
3250
00
3500
00
Prefix
Lo
C (
ms) PIC
no PIC
1
10
100
1000
10000
100000
1000000
0
5000
0
1000
00
1500
00
2000
00
2500
00
3000
00
3500
00
4000
00
4500
00
5000
00
Prefix
msec
250k PIC
250k no PIC
500k PIC
500k no PIC
PIC Core PIC Edge
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 60
Fast Restoration Design Take-Away
Leverage IP FRR (LFA) with MPLS / LDP wherever possible ‒ LFA is simpler, local (requires no
interoperability)
Leverage TE FRR, if we must have to. ‒ Bandwidth Engineering, for example.
Leverage BGP PIC for faster BGP convergence ‒ PIC is local (requires no interoperability)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 61
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Consideration
‒ Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Topological Consideration
What topology is chosen makes a big difference ‒ convergence, traffic engineering, capacity planning, routing table, stability..
Topological Options may vary ‒ Flat vs. Hierarchical
‒ Hub & Spoke vs. Ring (Square)
Also, the evergreen question about ECMP vs. LAG
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Flat PoP Topology LFA Applicability
Reference
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Hierarchical PoP Topology LFA Applicability
Reference
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Topological Consideration ECMP vs. Link-Bundle
Factors E-LinkBundle
ECMP
1 Member Links’ Speed Must Be Same Yes No
2 Member Links on Any LC Yes Yes
3 Routing Adjacency One Many
4 Routing Table Impact No Yes
5 Max Number of Member Links 64 16 (32*)
6 Line-Rate Multicast (Members on Any LCs) Yes? Yes
7 Port Failure Localized to the Router Yes No
8 BFD on Each Member Link Yes** Yes
9 Video Monitoring – Better Accounting No Yes
10 Non-Stop Routing (NSR), Forwarding (NSF) Yes Yes
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 66
Topological Consideration Take-Away
Triangle topology (i.e. Hub & Spoke) for PE connectivity is advantageous ‒ Naturally benefits from IP FRR
Linkbundling gaining more traction
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 67
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Consideration
‒ Test Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Results
The solution discussed here is a part of a complete end-to-end architecture for delivery of residential, business, and RAN backhaul services ‒ It is thoroughly validated for each service in the areas of:
- Functionality, Scalability, Performance / SLA, QoS, High Availability, Network Management, OAM
The solution (results) is well documented ‒ Design & Implementation Guide (DIG) available through your
account team
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
ASR-9000
CRS-1
7600
ASR-1000
3400E / 4948 MWR-2941 / ISR
Internet Video Headend/DC SEF Infrastructure
PoP A Hub & Spoke Aggregation Topology
PoP B Ring Aggregation Topology
PoP C Business MSE (Ethernet + TDM)
10GE 1GE
NGN Testbed
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
NGN Testbed – Platforms
Role Platform Version Aggregation Node
ASR-9000 IOS-XR 4.0.1
Core Node CRS-1/3 IOS-XR 4.0.1
Access Node ME-3400E 12.2(55)SE
Access Node 4500, 4948 15.0(2)SG
Access Node MWR-2941-A CSR 3.3
Service Edge Node
ASR-1000, Cisco 7600
15.1(1)S
Active Network Ab t ti
3.7.2
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Test Area Results
1 Topology Scalability PoPs – 100 (3 Real + 97 Simulated) Infrastructure BGP Routes – 100K; Infrastructure ISIS Routes – 12K;
2 Service Scalability Residential – 120K Triple Play Subscribers; Business L2VPN – 16K E-Line, 4K E-LAN (20K MACs); Business L3VPN – 4K VPNs (1M Routes); IP RAN/ TDM – 4K AToM PWE3;
3 Service (High) Availability Link & Node Failure and Recovery: <50 msec (Hub & Spoke Topology) <500 msec (Ring Topology)
Results – Summary
* A Few Exceptions During Node Recovery with High Scale
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Results (Just an Example) Service Convergence During Link Failure
0
100
200
300
400
500
600
700
800
NNI Failure (H&S)
NNI Failure (ring)
UNI Failure (Ethernet)
UNI Failure (uWave)
50
500
200
750
RAN Backhaul Service
RAN Backhaul Service
msec
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 73
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Consideration
‒ Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Case Study
SPs are fast embracing Cisco NGN reference
The next two slides illustrate the actual deployed networks -
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Case Study #1 APAC Mobile Operator / SP
PE PE
U-PE U-PE
Star
Regional Data Center
CSR CSR
mx40GE
SR SR
CR CR
PE PE
U-PE U-PE
CSR CSR
CR CR
mx40GE
nx10GE nx10GE
Mini-Core
Aggregation
Access
CSR - Cell Site Router SR – Service Router CR – Core Router BR – Backbone Router
300 Mbps per CSR (Radio)
9 Gbps per U-PE <10 CSR (Radios) per U-PE
3x40GE’s per SR Pair 378Gbps per SR Pair
Backbone
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Tier 1 Hub
Tier 2 Hub
SDC1 SDC 2
7600/ASR9k
Backbone
Distribution
Aggregation
Hub Agg
Hub Router
7600/ASR9k
ASR9k
Service Routers
Legend 1 GE Link
10 GE Ring Link
10 GE Point to Point Link
Video EQAM
CMTS
7600/ASR9k
Redundant SDC May Not Be Present
Case Study #2 US Cable Operator / SP
Tier 1 Hub ASR9k
CPEs
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSPG-2405 77
Introduction
Solution Overview ‒ Unicast Routing + MPLS Design
‒ Fast Restoration
‒ Topology Consideration
‒ Results
Case Study
Conclusion
Agenda
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Conclusion
Learned design options for large networks ‒ How to scale Routing (+MPLS) !
‒ What Fast Restoration technique is suitable! Where!
‒ Which Topology makes sense !
‒ Services Consideration !
Got the proof points ‒ Deployed case studies
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Additional Slides
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
LFA Roadmap – IPv4
MPLS TE-FRR 1-hop link
7600 (IOS)
ASR1000 (IOS-XE)
ASR9k (IOS-XR)
CRS-1 (IOS-XR)
Per Link LFA FRR Not Available Not Available 4.0.1 3.5.0
OSPF LFA FRR (per prefix)
15.1(3)S 3.4S 4.2.0 4.2.0
ISIS LFA FRR (per prefix)
15.1(2)S 3.4S 4.0.1 4.0.1
EIGRP FRR (per prefix) 15.2(4)S* 3.7S*
OSPF Remote LFA 15.2(2)S 3.6S 4.3.1* 4.3.1*
ISIS Remote LFA 15.2(2)S 3.6S 4.3.1* 4.3.1*
BGP PIC Core for IP/MPLS
12.2(33)SRC 2.5S 3.7.0 3.4
BGP PIC Edge 12.2(33)SRE 2.5S 4.0.0 4.0.0
*Future
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
LFA Roadmap – IPv6
MPLS TE-FRR 1-hop link
7600 (IOS)
ASR1000 (IOS-XE)
ASR9k (IOS-XR)
CRS-1 (IOS-XR)
Per Link LFA FRR Not Available Not Available 4.3.1* 4.3.1*
OSPF LFA FRR (per prefix)
Radar Radar 4.3.1* 4.3.1*
ISIS LFA FRR (per prefix) Radar Radar 4.3.1* 4.3.1*
EIGRP FRR (per prefix) Radar Radar Radar Radar
OSPF Remote LFA Radar Radar Radar Radar
ISIS Remote LFA Radar Radar Radar Radar
BGP PIC Core 3.5S 3.5S 3.7.0 3.4
BGP PIC Edge 3.5S 3.5S 4.0.0 4.0.0
*Future
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
BGP Next-Hop Tracking
Makes the next-hop failure detection event-driven instead of timer-driven
Next-hop tracking (NHT) feature allows to track BGP next-hops in the RIB
If the RIB entry changes, then the client such as BGP is notified
Allows for new path selection for BGP routes as soon as the notification is received
On/off knob as well as configuration option on how long to wait before starting new path selection
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Behavior Without NHT
Site2
RR1
P2
RR2
P4
P1 P3
PE1# show ip route 192.168.1.3 % Subnet Not in Table
PE1
PE3
PE4
pe1#sh ip bgp vpnv4 vrf vpna 10.1.2.0/24 BGP routing table entry for 100:1:10.1.2.0/24, version 42 Paths: (1 available, best #1, table vpna) Advertised to update-groups: 1 Local 192.168.1.3 (metric 145) from 192.168.1.2 (192.168.1.2) Origin incomplete, metric 0, localpref 100, valid, internal, best Extended Community: RT:100:1 Originator: 192.168.1.3, Cluster list: 192.168.1.2, mpls labels in/out nolabel/28
Tic…Tic…60sec
Wait 180 Seconds?? No!!!
Traffic Loss for Up to 60 Secs Due to BGP Scanner Interval
10.1.2.0/24
Site1 10.1.1.0/24
CE1
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
Site1 10.1.1.0/24
RR1 RR2
P2 P4
P1 P3
CE1 PE1
PE3
PE4
wg2pe1#sh ip bgp vpnv4 all 10.1.2.0 BGP routing table entry for 100:1:10.1.2.0/24, version 51 Paths: (1 available, best #1, table vpna) Flag: 0x820 Advertised to update-groups: 1 Local 192.168.1.4 (metric 193) from 192.168.1.2 (192.168.1.2) Origin incomplete, metric 0, localpref 100, valid, internal, best Extended Community: RT:100:1 Originator: 192.168.1.4, Cluster list: 192.168.1.2, mpls labels in/out nolabel/32
The Time Period Determines How Long BGP Will Wait Before Running the Best Path Algorithm After Notification Is Received.
router bgp 100 address-family ipv4 unicast bgp nexthop trigger enable bgp nexthop trigger delay 5
Potential Time Saving Is Up to 60 Secs
Site2
10.1.2.0/24
Edge FC
Edge FRR
POP FRR
Core FRR
POP FC Core FC Behavior with NHT-Enabled