17
Deploying 400Gbit/s on the Janet network Rob Evans, Chief Network Architect
Deploying 400Gbit/s on the Janet networkRob Evans, Chief Network Architect
What is Janet?
Deploying 400Gbit/s on the Janet network2
•National network for research and education
•Connects Higher and Further Education sites
•High speed connections to GÉANT, which connects all European R&E networks•…and on to other global NRENs
•”Everyday” use, students on their phones connected to wifi, through to large data flows from, e.g., the Large Hadron Collider, Bioinformatics, etc
•The question that everybody asks:•Jisc is the company •Janet is the network•Neither are acronyms any longer
What is Janet?
Deploying 400Gbit/s on the Janet network3
•Leased fibre from SSET
•Ciena 6500 transmission equipment managed in-house
•Predominantly Juniper routing base•MX2010•MX960
•Current network has been in service since 2013
•Fibre contract until 2028
•Previously regional aggregation networks managed autonomously
•Now all managed in-house•Process of rolling out new aggregation architecture to them all at the moment•Except Wales and Kent, which have all-sector public service aggregation networks
The Janet network – before.
Deploying 400Gbit/s on the Janet network4
The Janet network – the plan.
Deploying 400Gbit/s on the Janet network5
Why did we want/need 400G?
Deploying 400Gbit/s on the Janet network6
•Ever-increasing capacity demands• Incoming IP traffic to Janet from GEANT, peering, and transit is > 600Gbit/s
•More efficient spectrum usage
•Save on chassis space!•4x100G client plus 400G line in single slot, saves 3 slots compared to OTR (one slot per 100G), saves 7 slots (half a chassis) compared to OCI+OCLD (two slots per 100G).
•Flexgrid 9:1 WSS are two slots, older fixed grid 9:1 WSS occupied 3
Where should we deploy 400G?
7
•You might have thought where we wanted 600G•Busiest parts of the network•Lots of disruption to migrate from fixed grid to flexgrid.•Some routes too long to run Wavelogic Ai cards at 400G in 75GHz.• Other Wavelogic cards are available…
•Birmingham – Manchester – Leeds – Nottingham•Require 400G capacity•Replace existing 100G cards and re-deploy for South upgrade from 400G to 600G.
Will it work?
Deploying 400Gbit/s on the Janet network8
•100G transmission uses DP-QPSK encoding•Two polarisations, quadrature phase shifting
•Wavelogic Ai chipset adjusts encoding depending on OSNR•The better the OSNR, the more capacity•Proprietary modulation formats based on QPSK/8QAM/16QAM/32QAM/64QAM
•Step 1: Know what we’ve got•Original OTDR results now six years old•Take backbone fibres out of service to re-measure
OSNR: Why and how?
Deploying 400Gbit/s on the Janet network9
•Change from EDFA-based amplifiers to Raman•Strict loss requirements in the first 20km of fibre•Raman amplifiers include OTDR to check fibre is in spec before enabling
•Modelling on the planned spans•Good to go on Birmingham to Manchester and Nottingham to Leeds (& vice versa)•Manchester to Leeds: -0.42dB•Leeds to Manchester: -0.09dB•Anything below -0.3dB is out of spec•Conservative stance is that we can only do 300G between Leeds and Manchester
How does the fibre get from the road to the rack?
Deploying 400Gbit/s on the Janet network10
•Equinix MA1
Equinix Manchester
Incoming fibreEquinix
‘meet me’ rack
(basement)
Equinix ‘meet me’
rack(4th floor)
Janet optical
equipment rack
patch patch patch patchSupplier ‘meet me’
rack
Janet patching
rack
•Just splice the thing…• Spot the deliberate mistake in the diagram...
Flexgrid and colourless add/drop
Deploying 400Gbit/s on the Janet network11
•Existing WSSs all 50GHz fixed grid with wavelength-specific add/drop mux/demux (CMD44)
•400Gbit/s uses 56Gbaud in 75GHz
•‘Simple’ matter of replacing fixed-grid WSSs with flex-grid and introducing colourless add/drop.
Flexgrid conversion
Deploying 400Gbit/s on the Janet network12
Not quite colourless
Deploying 400Gbit/s on the Janet network13
OSC SFP 0-34dB Low TxPEC: NTK592NG
Loc: Shelf:3 Slot:14.2
Out
In
Line A In Line A OutA
Line B OutB
Line B In
LCLC6
7
5
8
6500 MLA2PEC: NTK552FA
Loc: Shelf: 3 Slot: 10
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
6500
WS
S 50 G
Hz w
/OP
M 1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 3 Slot: 11
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Switch1 Out
Switch1 In
4
3
Swi tch3 Out
Switch3 In
8
7
Swi tch4 Out
Switch4 In
10
9
Switch5 Out12
Switch5 In11
Swi tch6 Out14
Switch6 In13
Swi tch7 Out16
Switch7 In15
Swi tch8 Out18
Switch8 In17
Swi tch9 Out20
Switch9 In19
Switch2 Out
Switch2 In
6
5
12 dB 12
Out
In
SC
To:U
nn
w_
man
ch
NE
219
7-1
-1
Line A In Line A OutA
Line B OutB
Line B In
LCLC
6
7
5
8
6500 MLAPEC: NTK552BA
Loc: Shelf: 1 Slot: 10
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
6500
WS
S 5
0 GH
z w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 1 Slot: 11
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Swi tch1 Out
Switch1 In
4
3
Swi tch3 Out
Switch3 In
8
7
Swi tch4 Out
Swi tch4 In
10
9
Switch5 Out12
Swi tch5 In11
Swi tch6 Out14
Swi tch6 In13
Swi tch7 Out16
Switch7 In15
Swi tch8 Out18
Swi tch8 In17
Swi tch9 Out20
Switch9 In19
Switch2 Out
Swi tch2 In
6
5
BMD2PEC: NTT862DA
Loc: Shelf:1 Slot:83
CommonOut
1Common
In
2
Input 1
Output 14
3
Input 2
Output 26
5
eCMD44 50GHz Red C-Band with TAPPEC: NTT862BD Shelf: 1 Slot: 85
89
90
92
6 dB1 2
OSC SFP 0-34dB Low TxPEC: NTK592NG
Loc: Shelf:1 Slot:14.1
Out
In
Line A InLine A OutA
Line B OutB
Line B In
LC LC
6
7
5
8
6500 MLA2PEC: NTK552FA
Loc: Shelf: 1 Slot: 9
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
6500
WS
S 5
0 G
Hz
w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 1 Slot: 6
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Switch1 Out
Switch1 In
4
3
Swi tch3 Out
Switch3 In
8
7
Swi tch4 Out
Switch4 In
10
9
Switch5 Out12
Switch5 In11
Swi tch6 Out14
Switch6 In13
Swi tch7 Out16
Switch7 In15
Swi tch8 Out18
Switch8 In17
Swi tch9 Out20
Switch9 In19
Switch2 Out
Swi tch2 In
6
5
Out
In
SC
To:A
ched
dle
NE
46
79-1
-2
OSC SFP 0-34dB Low TxPEC: NTK592NG
Loc: Shelf:2 Slot:14.1
Out
In
Line A InLine A OutA
Line B OutB
Line B In
LC LC
6
7
5
8
6500 MLA2PEC: NTK552FA
Loc: Shelf: 2 Slot: 9
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
6500
WS
S 5
0 G
Hz
w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 2 Slot: 6
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Switch1 Out
Switch1 In
4
3
Switch3 Out
Switch3 In
8
7
Switch4 Out
Swi tch4 In
10
9
Switch5 Out12
Swi tch5 In11
Switch6 Out14
Swi tch6 In13
Switch7 Out16
Switch7 In15
Switch8 Out18
Swi tch8 In17
Switch9 Out20
Switch9 In19
Swi tch2 Out
Switch2 In
6
5
Out
In
SC
To:S
outh
port
NE
227
9-4
-1
Line A InLine A OutA
Line B OutB
Line B In
LC LC6
7
5
8
6500 MLAPEC: NTK552BA
Loc: Shelf: 2 Slot: 10
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
6500
WS
S 5
0 G
Hz
w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 2 Slot: 11
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Switch1 Out
Switch1 In
4
3
Switch3 Out
Switch3 In
8
7
Switch4 Out
Switch4 In
10
9
Switch5 Out12
Switch5 In11
Switch6 Out14
Switch6 In13
Switch7 Out16
Switch7 In15
Switch8 Out18
Switch8 In17
Switch9 Out20
Switch9 In19
Swi tch2 Out
Switch2 In
6
5
BMD2PEC: NTT862DA
Loc: Shelf:2 Slot:83
CommonOut
1Common
In
2
Input 1
Output 14
3
Input 2
Output 26
5
eCMD44 50GHz Red C-Band with TAPPEC: NTT862BD Shelf: 2 Slot: 85
89
90
92
6 dB 12
6500
WS
S 5
0 GH
z w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 3 Slot: 6
LCLC
OPMMonit or 2
1
2
Monit or 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Swi tch1 Out
Switch1 In
4
3
Switch3 Out
Switch3 In
8
7
Switch4 Out
Switch4 In
10
9
Switch5 Out12
Switch5 In11
Switch6 Out14
Switch6 In13
Switch7 Out16
Switch7 In15
Switch8 Out18
Switch8 In17
Switch9 Out20
Switch9 In19
Swi tch2 Out
Swi tch2 In
6
5
Line A In Line A OutA
Line B OutB
Line B In
LCLC
6
7
5
8
6500 MLA2PEC: NTK552FA
Loc: Shelf: 3 Slot: 9
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LCOSC SFP 0-34dB Low Tx
PEC: NTK592NGLoc: Shelf:3 Slot:14.1
Out
In
Out
In
SC
To:Apre
son
NE
550
5-1-1
OSC SFP 0-34dB Low TxPEC: NTK592NG
Loc: Shelf:5 Slot:1.1
Out
In
Line A In Line A Out
Line B Out Line B In
LCLC
6
7
5
8
6500 MLA3PEC: NTK552GAE5
Loc: Shelf: 5 Slot: 5
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
6500
WS
S 5
0 GH
z w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 5 Slot: 2
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Swi tch1 Out
Switch1 In
4
3
Swi tch3 Out
Switch3 In
8
7
Swi tch4 Out
Swi tch4 In
10
9
Switch5 Out12
Swi tch5 In11
Swi tch6 Out14
Swi tch6 In13
Swi tch7 Out16
Switch7 In15
Swi tch8 Out18
Swi tch8 In17
Swi tch9 Out20
Switch9 In19
Switch2 Out
Swi tch2 In
6
5
Out
In
SC
To:D
ublin
NE
-1-2
PEC: NTK553FAShelf: 3 Slot: 3 OTS: 1DIA: True
9
10
OSC SFP 0-34dB Low TxPEC: NTK592NG
Loc: Shelf:4 Slot:14.1
Out
In
Line A InLine A Out A
Line B OutB
Line B In
LC LC
6
7
5
8
6500 MLAPEC: NTK552BA
Loc: Shelf: 4 Slot: 10
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
6500
WS
S 5
0 G
Hz
w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 4 Slot: 11
LCLC
OPMMonit or 2
1
2
Monit or 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Switch1 Out
Swi tch1 In
4
3
Switch3 Out
Swi tch3 In
8
7
Switch4 Out
Switch4 In
10
9
Swi tch5 Out12
Switch5 In11
Switch6 Out14
Switch6 In13
Switch7 Out16
Swi tch7 In15
Switch8 Out18
Switch8 In17
Switch9 Out20
Swi tch9 In19
Switch2 Out
Switch2 In
6
5
13 dB1 2
Out
In
SC
To:CP
LR_
TM
an
ch_
Wre
xN
E1
841
9-1
-1
WS
S 5
0G
Hz
w/O
PM
9x1
2-s
lot-
wid
e
LC
LC
LC
OME6500 WSSPEC: NTK553FC
Shelf: 7 Slot: 3
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Swi tch1 Out
Switch1 In
4
3
Swi tch3 Out
Switch3 In
8
7
Swi tch4 Out
Swi tch4 In
10
9
Switch5 Out12
Swi tch5 In11
Swi tch6 Out14
Swi tch6 In13
Swi tch7 Out16
Switch7 In15
Switch8 Out Drop118
Switch8 In Add117
Swi tch9 Out Drop220
Switch9 In Add219
Switch2 Out
Switch2 In
6
5
Line B In Line B OutB
Line A OutA
Line A In
LCLC
8
5
7
6
MLA2PEC: NTK552FAE5
Loc: Shelf:7 Slot:2
Monitor 1
Monit or 2
LC
2
1
OSC B In
OSC A Out
3
4
LCOSC SFP 0-34dB Low Tx
PEC: NTK592NGLoc: Shelf:7 Slot:1.1
Out
In
In
Out
SC
To:J
od
rell B
ank
ma
ccjb-dm
x1-1
-1
CMD 6489
90
92
6500
WS
S 5
0 G
Hz
w/O
PM
1x9
LC
LC
LC
6500 WSS 50 GHz 1x9PEC: NTK553FA
Shelf: 1 Slot: 5
LCLC
OPMMonitor 2
1
2
Monitor 1LC
LC
LC
LC
LC
LC
Common In
Common Out
21
22
Switch1 Out
Switch1 In
4
3
Switch3 Out
Switch3 In
8
7
Switch4 Out
Switch4 In
10
9
Switch5 Out12
Switch5 In11
Switch6 Out14
Switch6 In13
Switch7 Out16
Switch7 In15
Switch8 Out18
Switch8 In17
Switch9 Out20
Switch9 In19
Switch2 Out
Switch2 In
6
5
Line A InLine A Out A
Line B OutLine B In
LC LC6
7
5
8
6500 SLAPEC: NTK552AA
Loc: Shelf: 1 Slot: 4
Line B Mon
Line A Mon
LC
2
1
OSC B In
OSC A Out
3
4
LC
If only it was that easy…
Deploying 400Gbit/s on the Janet network14
•Network-wide software upgrade beforehand•Not just one, but two: 10.2 -> 11.2 -> 12.1•OneControl
•Change in licensing approach for 12.1•License alarms after upgrade•Deploy external license server
•Different power calculations between releases of software•Existing services entered a “waiting for power” state after upgrade
It was all worth it in the end…
Deploying 400Gbit/s on the Janet network15
•Initially provisioned Manchester – Leeds with 3x100G clients•Wavelength came up at 400G•Able to add fourth service and run it at 400G
Where next?
Deploying 400Gbit/s on the Janet network16
•Migrating the rest of the network to flexgrid•Just completed
•Deploying true colourless add/drop•Next up
•Newer cards for the longer spans
•400GE clients•A whole other talk… on QSFP56-DD, etc
Questions?
Deploying 400Gbit/s on the Janet network17
