The intersection of optical transport and routing in next-
generation networksPhil Bedard – TME@Cisco
NANOG7710/30/2019
Agenda• Current multi-layer network architecture
• Design • Issues
• Next-generation architecture• Design• Benefits • Enabling technologies• Benefits
• Modeling example
Topology Evolution for SP Networks
Traffic Demand
• Ring based networks based on physical fiber
• Optical transport was not very advanced, so either P2P IP links or, channelized over TDM, or ATM overlay
• 4 interfaces to upgrade, 10 for redundancy
• Fiber still ring in many cases, but L1 circuit topology dictated by traffic demands
• Why did this happen? • Router interfaces were very expen$ive• Optical transport is less expensive? • Pace of optical channel innovation allowed better
utilization of fiber resources
A
Z
A
Z
“Hop by Hop” Bypass
Traffic Demand
Current Multi-Layer Networks
• Active transport elements with ROADMs and muxponders
• Transport and packet networks isolated using gray optics
• Most common deployment in high speed regional and backbone networks
• 1G/10G CWDM/DWDM optics in routers
• 100G/200G using integrated optics, ACO, or DCO optics, typically CFP2 or shorter reach with SFP+, QSFP28
• Most popular in DCI and access networks
Router + Muxponder IPoWDM
Connectivity Types – Linear View
• Standard optics based on distance
• Single network to manage
• BW per fiber limited to single optic BW
• Grey optics between router and muxponder, coherent output from muxponder
• Clients 1:1 mapping to wavelength or multiplexed onto 200G,400G,600G, 800G, 1Tbps channels
• 100-150W power dissipation per combined ”port”
• Minimum 5 optics per circuit
• No transponder required
• Requires channel mux
• 15-21W Power dissipation per port
• Interop with ROADM, maximize use of existing infrastructures
• 2 optics per circuit
IP+DWDMDark Fiber IPoWDM
Issues with current IP+Optical deployments • Multiple opaque networks
• CapEx, OpEx cost of multiple networks • Difficult to share information such as SRLG, GMPLS was not successful• Transport control-plane is proprietary • Circuit deployment complexity across networks
• Different protection schemes at both layers of the network• Difficult to coordinate between layers• Network state difficult to sync after optical protection/restoration
• Organizational silos• Planning • Engineering • Operations
”Router” Group
“Transport” Group
Existing Multi-Layer Topology
High speed IP network – No OTN services/switching
New Single-Layer Single-Hop Fabric
Single-Layer Transport Elements
• C-band and L-band passive multiplexers • Local amplifiers and ILA
No complex CDC add/drop, IP is any to anyNo multi-degree ROADMs, every wavelength terminates at every IP node No regen with OEO at every junction (distancedependent)
Haven’t we been here before? • “Multilayer Convergence” has been talked about for many years
• CRS with 40G IPoDWDM line cards in 2007, many other vendor initiatives, TIP Voyager, etc.
• Becoming more popular in access and aggregation with CFP2-DCO 100G/200G
• Why hasn’t this previously become mainstream for most networks? • Integrated optics were expensive and locked ports into being only DWDM • Optical channel tech has outpaced router electronics and standards • Density, density, density (and cost)
Benefits
• IP layer FRR is used for all traffic protection driven by optical performance (pre-FEC)
• Overall IP network utilization• Wavelength efficiency • Simplifies network operation• Similar or lower CapEx spend • Faster time to turn up
Does not solve
• Still need wavelength planning
• Still need to manage some photonic equipment
• TDM, OTN, and wavelength services need emulation for transparency
Wavelength Utilization
• ROADM bypass requires longer optical links
• Longer links supports lower bit-rates, reducing wavelength (and consequently L0 infra) utilization
• Hop-by-Hop uses more L3 I/Fs, but optimizes fiber utilization
ROADM by-passOptical Length
1,600 km
Hop-by-HopOptical Length
400 km
L0 Forwarding(ROADM By-pass)
L3 Forwarding
400 km
200G
400G
50% Util.
100% Util.
Innovations enabling the architecture
100G/200G CFP2-DCO now for access and aggregation
400G ZR+ using CFP2-DCO and QSFP56-DD pluggable transceiversQSFP56-DD = No density tradeoff
Router NPU bandwidth capacity
Continued reduction in IP router interface costs
Complexity in optics will drive minimization of transceiver counts
Coherent router optics evolution
5x7 inches
3x6 inchesCFP2-ACO
2011 2014 2016 2019 2020
QSFP-DD DCO
CFP2-DCO
More integration
400G Standards Reference
3 m
400G-CR88x 50G-CR
100 m
400G-SR8400G-SR4.2400G-AOC
0.5-2 km
400G-DR4400G-FR4
10 km
400G-LR4400G-LR8
20+ km
OIF 400ZROpen 400ZR+
Copper Cables
MMF / AOC
SMFDuplex
SMFDuplex
SMFDuplex
Dist
ance
Opt
ics
Med
ia
ZR/ZR+ are coherent DWDM optics, ZR+ >15W
400G ZR OIF and Open ZR+FORM FACTOR QSFP56-DD – No Density Penalty CFP2-DCO
TECHNOLOGY ZR OpenZR+ ZR Open ZR+
BW SUPPORT400G
(Single Fixed Wavelength)
400G/300G/200G/100G(Tunable)
400G (Single Fixed Wavelength)
400G/300G/200G/100G (Tunable)
Tx POWER -10 -10/-9/-6/-6 -10 +3
UNAMPLIFIED REACH 400G: 20km
400G: 40km300G: 40km200G: 80km100G: 80km
400G: 40km
400G: 100km300G: 112km200G: 124km100G: 124km
AMPLIFIED REACH 400G: 80km
400G: 1300km300G: 2500km200G: 2500km100G: 2500km
400G: 80km
400G: 1300km300G: 2500km200G: 2500km100G: 4000km
POWER DISSIPATION 15 15-21 15 15-21
FEC C-FEC oFEC, SC-FEC for 100G C-FEC oFEC, SC-FEC for 100G
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2015 2016 2017 2018 2019 2020 2021
Router capacity and economics
14M
12M
10M
8M
6M
4M
2M
Gig
abits
NPU
(Mbps)
300B
250B
200B
150B
100B
50B
Gigabits NPU Bandwidth
Internet Gigabits per Month vs. NPU BW
Platform HWOptics
Optics vs. Host Interface
WHY? Host port costs are decreasing FASTER than optics technologies. Optics complexity increases with speed.
Pluggable Optics
Router Interface
100%
0%Speed
Total Cost
10G 100G 400G
Example Regional Network Topology
• 55 Sites• ~100 km across• Two Cache Sites• Two Internet Gateways• Cache and Gateway sites also support local subscribers
Traffic Model
• Edge site traffic scaled to site weight• Site weight dynamic range 60:1
• 75% of traffic to cache sites, 25% to gateways• Each edge site connected diversely to gateways and to cache sites
Hop-by-Hop Approach
• Single-layer one-hop end to end • Single hop transmission negates need for regeneration• Some spans require large LAGs
• Largest LAG 29 x 400G for illustrated example of 50 Tbps load
1330 x 400G Interfaces
Hollow Core Approach
• One-to-one mapping of trafficdemands to end-to-end wavelengths
• Longer reach wavelengths feature B2B regen• Fewer, smaller LAGs (Largest 3 x 400G)
794 x 400G Interfaces(one third as regen)
Optimized Bypass Approach
• Bypasses kept below reach limitto avoid regeneration
• Largest LAG 11 x 400G506 x 400G Interfaces
Comparing The Models
Hop-by-Hop Hollow Core Bypass
ROADM Layer No Yes Yes
Reach Sensitivity
None High Low
Interfaces 1330 794 506
Utilization 95% 45% 82%
Largest LAG 29 3 11
Max Waves/Span
29 58 30
Adjacencies 79 216 132
Load:50 Tbit/s
Reach:5 Hops
Contrasting Approaches As Traffic Scales
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0
250
500
750
1000
1250
1500
1750
2000
2250
2500
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
55000
60000
65000
70000
75000
80000
85000
90000
95000
100000
U"lisa"
on
Interfaces
Load(Gbit/s)
H-by-H Bypass HollowCore HHU<l BPU<l HollowU<l
400G
Q & A
400G DCO Pluggable - Options
• Supported Trunk Rate:• 200G, 300G and 400G (AT&T OpenFEC)• OTU4 with Staircase FEC
• Channel Spacing:• 75GHz (Min)• OTU4: 50GHz (Min)
• TX Power Range: • -10 to +1dBm (SW Configurable)
• RX Sensitivity:• -19dBm• OTU4: -22dBm
• CD Compensation:• OTU4: +/-20,000ps/nm• 200G: +/-50,000ps/nm• 300G: +/-30,000ps/nm• 400G: +/-16,000ps/nm
• DGD Compensation:• 200G, 300G, 400G: 60ps• OTU4: 90ps
• Supported Trunk Rate:• 200G, 300G and 400G (AT&T OpenFEC)• OTU4 with Staircase FEC
• Channel Spacing:• 75GHz (Min)• OTU4: 50GHz (Min)
• TX Power Range: • -10dBm
• RX Sensitivity:• -19dBm• OTU4: -22dBm
• CD Compensation:• OTU4: +/-20,000ps/nm• 200G: +/-50,000ps/nm• 300G: +/-30,000ps/nm• 400G: +/-16,000ps/nm
• DGD Compensation:• 200G, 300G, 400G: 60ps• OTU4: 90ps
400G CFP2 DCO
400G DD-QSFP56 ZR/ZR+ DCO