Reducing the Energy Consumption of Photonics Hardware in Data Center Networks
Richard Penty, Jonathan Ingham, Adrian Wonfor, Kai Wang, Ian White
1
Richard Penty, Jonathan Ingham, Adrian Wonfor, Kai Wang, Ian White
Centre for Photonic Systems, Electrical Engineering Division, Engineering Department, University of Cambridge, 9 JJ Thomson Avenue, Cambr idge CB3 0FA UK
• Introduction to Short Link Systems
• Growing energy requirements in the internet
• Use of optics to reduce supercomputer/data center energy cost
• Modulation formats for next -generation optical
Key Themes
2
• Modulation formats for next -generation optical datacommunication links
• Candidate modulation formats
• Carrierless amplitude and phase modulation
• (If time) Reducing energy consumption of optical sw itches
• Active-passive OEICs
• Conclusions
Tota
l Pow
er C
onsu
mpt
ion
(W)
1012Global electricity supply
1013
3% p.a.
Power Consumption of Internet
Power Consumption of the Global InternetTo
tal P
ower
Con
sum
ptio
n (W
)
1.5 billion users
Year2010 2015 2020 2025109
1011
1010
40% p.a. Data growth
10% p.a. Growth in user numbers
Power Consumption of Internet
Sources: Hinton et al., Tucker, IEEE
Comparative Energy Efficiency in Photonics
Fibre Channel Speed RoadMap
5http://www.fibrechannel.org/roadmaps
Evolution of Datacom Standards - Infiniband
6Infiniband Roadmap
Optical Interconnects
J. Bautista, Optoelectronic Integrated Circuits Vii, pp. 1-8, 2005.
7
2005.
• Optical interconnects offer significant advantages over their electrical counterparts:
- large link bandwidth, reduced power consumption, EMI, thermal management issues
- but users will only use optics if they have to
- photonics costs in for longer reach and higher bandwidthc
Parallel Optical Interconnects in Supercomputers
by IBM
Copper Replacement by VCSELs and Fibers:
a) IBM Roadrunner (2008), 1 Petaflop: Fiber to the Rack; 50,000 optical links.b) IBM Blue Waters (2011), 10 Petaflops: Fiber to the Module; 5 Million optical
links.
• 30 MW power consumption of the lasers in the optical links in Exaflop computers
• ~24 MW cooling system
• +receiver, electronics
Record energy-to-data ratio (EDR) of
83 fJ/bit at 25°C and
81 fJ/bit at 55°C and heat-to-bit rate ratio (HBR) of 69 mW/Tbps at 17 Gb/s across 100 m fiber
Record Energy Efficiency
1 Terabit per second for less than 100 mW
• Introduction to Short Link Systems
• Growing energy requirements in the internet
• Use of optics to reduce supercomputer/data center energy cost
• Modulation formats for next -generation optical
Key Themes
11
• Modulation formats for next -generation optical datacommunication links
• Candidate modulation formats
• Carrierless amplitude and phase modulation
• (If time) Reducing energy consumption of optical sw itches
• Active-passive OEICs
• Conclusions
• Several possibilities for modulation format:• Non-return-to-zero modulation (NRZ)
Simple scheme, but symbol rate equal to bit rate, which requires high bandwidth lasers and receivers and dispersion effects are significant
• Multilevel modulation (e.g. PAM4)
In PAM4, the symbol rate is one half of the bit rate, therefore less demand placed on laser and receiver bandwidth and reduced dispersion effects compared to NRZ
Modulation formats for next-generation datacommunication links
12
laser and receiver bandwidth and reduced dispersion effects compared to NRZ
• Subcarrier modulation (SCM)
Allows high spectral efficiency due to use of multiple FDM channels on RF carrier frequencies. Needs RF components
• Orthogonal frequency division multiplexing (OFDM)
Essentially SCM with increased spectral efficiency. However, transmitter and receiver electronics complex, possibly with high power consumption
• Carrierless amplitude and phase modulation (CAP)
Flexible scheme with FDM channels, similar to SCM, but simpler electronics, with low power consumption and ability to operate as PAM if needed
NRZ PAMn Partial-Response CAP CDMA SCM (QAM) OFDM or ODMT
Carrierless Schemes Carrier or Multi-Carrier Schemes
1 Dimensional Schemes Multi-Dimensional Schemes
Spectrum of modulation formats
13
General directional of increasing bandwidth efficiency
General directional of increasing optical modulation power penalty
General directional of increasing electrical complexity and power dissipation
Recent results: 40 Gb/s NRZ
• Draka, Netherlands & HHI, Berlin
• NRZ modulation
• 1300 nm CW external-cavity diode laser & MZ modulator
• Specially-optimized 50-µm-core-diameter MMF
• Center launch or radially-overfilled launch investigated
• 40 Gb/s over 600 m of MMF achieved
14
• 40 Gb/s over 600 m of MMF achieved
P. Matthijsse et al., OFC 2006, paper OWI13
Recent results: 1.25 Gb/s PAM -4
Munich University of Technology & Technical University Eindhoven
PAM-4 modulation
Step-index polymer optical fiber (SI-POF)
1.25 Gb/s over 75 m SI-POF using a LED and predistorted PAM-4
15F. Breyer et al., ECOC 2008, paper We.2.A.3
Recent results: 37 Gb/s QAM -16
Chalmers University, Sweden
QAM-16 constellation using a “single-cycle” approach
Directly-modulated 850 nm VCSEL
10 Gbaud symbol rate, requiring less than 20 GHz modulation bandwidth
FEC with 7% overhead required
37.2 Gb/s over 200 m of OM3+ MMF achieved
16
0 6 12 18 24 30Frequency (GHz)
Pow
er (
5 dB
/div
)
200 m
In-phase
Qua
drat
ure
-6 -5 -4 -3 -2 -1 0 1
-6
-5
-4
-3
-2
Received optical power (dBm)
log(
BE
R)
BTB100 m200 m
Bit-error-rate @ 40 Gb/s
Electrical spectrum
K. Szczerba et al., ECOC 2010, paper We.7.B.2
Recent results: 8 Gb/s OFDM
Bell Labs, Stuttgart
OFDM modulation with 272 QPSK subcarriers
Modal diversity and MIMO processing employed
8 Gb/s over 5 km of 50-µm-core-diameter MMF achieved
But significant DSP required
17B. Franz et al., ECOC 2010, paper Tu.3.C.4
List of modulation schemes considered for 20 Gb/s MMF links20 Gb/s NRZ850 nm wavelength over OM3 fibre
20 GHz Tx/Rx parameters**
20 Gb/s PAM-4850 nm wavelength over OM3 fibre
10 GHz Tx/Rx parameters*
20 Gb/s duobinary850 nm wavelength over OM3 fibre
20 GHz Tx/Rx parameters**
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1time / ns
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1time / ns
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2time / ns
0
0.5
1
1.5
2
0 5 10 15 20-140
-120
-100
-80
-60
-40
-20
0
20
GHz
dB
o
40
QPSK
QAM-16
18
20 GHz Tx/Rx parameters**
* “10 GHz Tx/Rx parameters” indicates 35 ps Tx rise time (20% t o 80%) with a 7.5 GHz 4 th-order Bessel-Thomson LPF at Rx** “20 GHz Tx/Rx parameters” indicates 23.5 ps Tx rise time (2 0% to 80%) with a 15 GHz 4 th-order Bessel-Thomson LPF at Rx
20 Gb/s QPSK (4 x 5 Gb/s channels)850 nm wavelength over OM3 fibre
20 GHz Tx/Rx parameters**
20 Gb/s QAM-16 (20 x 1 Gb/s channels)850 nm wavelength over OM3 fibre
20 GHz Tx/Rx parameters**
0 0.01 0. 02 0. 03 0.04 0. 05 0.06 0.07 0.08 0.09 0.1
ns
0 5 10 15-120
-100
-80
-60
-40
-20
0
20
GHz
dB
o
Trade-offs must be considered in terms of link length, receiver sensitivity and complexity of implementation
QPSK and QAM offer increased spectral efficiency but with greater requirements in terms of SNR, linearity of the optical source and complexity of the associated electronics
Power budget comparison
Unallocated penalties (dBo) are shown in brown. This assumes a total power budget of 8 dBo for 850 nm links and 12 dBo for 1300 nm links
RIN penalty (dBo) is quoted at the 99th percentile of the installed base of MMF and shown in yellow. A laser source with RIN = –135 dB/Hz is used for all cases to enable comparison. This bar saturates at 5 dBo
Dispersion penalty (dBo) is quoted at the 99th percentile of the installed
19
U F D
Dispersion penalty (dBo) is quoted at the 99th percentile of the installedbase of MMF and shown in light blue. This bar saturates at 10 dBo
Relative receiver sensitivity (dBo) is shown in dark blue. The receivershave equal thermal noise power spectral density. The number may beviewed as a degradation in sensitivity relative to an unequalised receiverwith a sensitivity of –18 dBm at 10.3125 Gb/s with LRM Tx and Rx filtering
The left-hand bar corresponds to: unequalised receiver “U”
The central bar corresponds to: receiver with a 7-tap FFE “F”
The right-hand bar corresponds to: receiver with a (7, 5)-tap FFE-DFE “D”
Power budgets at 100 mNRZ PAM-4 Duobinary
RIN
UNALLOCATED
PENALTIES
Key
SCM
10
15
dBo
QP
SK
QA
M-1
6
20
U F D
1300 nm
20 GHz
U F D
850 nm
20 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
20 GHz
U F D
850 nm
20 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
15 GHz
U F D
1300 nm
15 GHz
RELATIVE
RECEIVER
SENSITIVITY
DISPERSION
PENALTY
RIN
PENALTY
U UNEQUALISED
F FFE
D FFE-DFE
U U
1300 nm
20 GHz
0
5
dBo
Power budgets at 200 mNRZ PAM-4 Duobinary
RIN
UNALLOCATED
PENALTIES
Key
SCM
QP
SK
QA
M-1
6
10
15
dBo
21
U F D
1300 nm
20 GHz
U F D
850 nm
20 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
20 GHz
U F D
850 nm
20 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
15 GHz
U F D
1300 nm
15 GHz
RELATIVE
RECEIVER
SENSITIVITY
DISPERSION
PENALTY
RIN
PENALTY
U UNEQUALISED
F FFE
D FFE-DFE
U U
1300 nm
20 GHz
0
5
dBo
Power budgets at 300 mNRZ PAM-4 Duobinary
RIN
UNALLOCATED
PENALTIES
Key
SCM
QP
SK
QA
M-1
6
10
15
dBo
22
U F D
1300 nm
20 GHz
U F D
850 nm
20 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
20 GHz
U F D
850 nm
20 GHz
U F D
1300 nm
10 GHz
U F D
850 nm
10 GHz
U F D
1300 nm
15 GHz
U F D
1300 nm
15 GHz
RELATIVE
RECEIVER
SENSITIVITY
DISPERSION
PENALTY
RIN
PENALTY
U UNEQUALISED
F FFE
D FFE-DFE
U U
1300 nm
20 GHz
0
5
dBo
50
CAP technique is highly flexible
Generation of passband channels, similar to subcarrier modulation, may be achieved through a simple change in the tap coefficients of an electronic filter
Avoids the requirement for upconversion using a mixer and local oscillator
Introduction to CAP
23
0 0.05 0.1 0.15 0.2ns
Eye diagram for one channel of a 40 Gb/s CAP16 system
0 10 20 30 40-100
-50
0
GHz
dBo
Corresponding electricalspectrum
T T T
Σ
…Input
Output
× × × ×c–L c1–L c2–L cL
T T T
Σ
…Input
Output
× × × ×c–L c1–L c2–L cL0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09
0
0.2
0.4
0.6
0.8
1
time / ns
ampl
itude
/ no
rmal
ised
line
ar u
nits
Flexible generation of CAP symbols
Electronic transversal filters developed for dispersion compensation
24
Use these filters for generating CAP pulses
Biphase shaping filter
Gb(f )
Modified-biphase shaping filter
G (f )
Block diagram of a 40 Gb/s CAP16 link
PRBS
27 – 1
20 Gb/s
Bit to symbol mapping
10 Gbaud
LPF
GaussianΣPRBS
27 – 1
Decorrelated
Bit to symbol mapping
10 Gbaud
25
Gmb (f )
SMF model
802.3ae
LPF
4th-order BT
Matched filter
Gb*(f ) or G mb
* (f )
SER calculation
Penalties at
BER = 10–12
Decorrelated
20 Gb/s10 Gbaud
40 Gb/s Eye Diagrams - CAP16 v. NRZ
0 0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05
NRZ
5 km 10 km 15 km 20 km
0 0.01 0.02 0.03 0.04 0.05ns
0 0.01 0.02 0.03 0.04 0.05ns
0 0.01 0.02 0.03 0.04 0.05ns
0 0.01 0.02 0.03 0.04 0.05ns
0 0.05 0.1 0.15 0.2ns
0 0.05 0.1 0.15 0.2ns
0 0.05 0.1 0.15 0.2ns
0 0.05 0.1 0.15 0.2ns
CAP16
40 Gb/s CAP16 – dispersion penalties compared to NRZ
6
8
10
12
disp
ersi
on p
enal
ty /
dB
o
Performance comparison with 40 Gb/s NRZ at 1550 nm
40 Gb/s CAP16 40 Gb/s NRZ
6
8
10
12
disp
ersi
on p
enal
ty /
dB
o
27
0 1 2 3 4 5 6 7 80
2
4
link length / km
disp
ersi
on p
enal
ty /
dB
o
Tx: 16.8 ps Tx 20% – 80% rise time (Gaussian LPF)
SMF: 1550 nm 17 ps / (nm km) (802.3ae model)
Rx: 21 GHz Rx –3 dBe bandwidth (4th-order Bessel-Thomson LPF)
In excess of 10 km at 1550 nm with CAP
0 5 10 15 20 25 300
2
4
link length / km
disp
ersi
on p
enal
ty /
dB
o
40 Gb/s Power Budgets - CAP16 v. NRZ
28
Σ
Q
I
DATA
DATA 6 dB
PR
BS
DC
AΣ Σ Σ Σ
TRANSVERSAL FILTER
TRANSVERSAL FILTER
TRANSVERSAL FILTER
BIPOLAR DATA MULTILEVEL DATA CAP MODULATOR
Σ
Q
I
DATA
DATA 6 dB
PR
BS
DC
AΣ Σ Σ Σ
TRANSVERSAL FILTER
TRANSVERSAL FILTER
TRANSVERSAL FILTER
BIPOLAR DATA MULTILEVEL DATA CAP MODULATOR
40 Gb/s CAP16 Experimental Demonstration
29
DC
A
: RF phase shifter
FILTER
CAP DEMODULATOR
DC
A
: RF phase shifter
FILTER
CAP DEMODULATOR
“40 Gb/s Carrierless Amplitude and Phase Modulation for Low-Cost Optical Datacommunication Links”
J. D. Ingham, R. V. Penty, I. H. White, D. G. Cunningham, OFC 2011
Encoded 40 Gb/s CAP16 eye diagram Decoded 40 Gb/s CAP16 eye diagram
Example results in datalinks: 40 Gb/s CAP-16
In phase
Back to back 10 km SMF
Carrierless amplitude and phase modulation
Low-power transversal-filter implementation
40 Gb/s over 10 km of SMF achieved
30
In phase
channel
20 ps/div20 ps/div
Quadrature
channel
0 0.05 0.1 0.15 0.2ns
Eye diagram for one channel of a 40 Gb/s CAP16 system
0 10 20 30 40-100
-50
0
50
GHz
dB
o
Corresponding electrical spectrum J. D. Ingham et al., OFC 2011, paper OThZ3
Can we combine PAM and CAP Codes?
• PAM and CAP codes – Complementary frequency distribu tions
– PAM-4 and CAP-4 at 10 Gb/s
– PAM-4 and CAP-2 at 13.3 Gb/s
Lane 1
Lane 2
Lane 3
λλλλ1
3 ×××× 13.3 Gb/s Lanes 40 Gb/s by 1-wavelength CWDMLane 1
Lane 2
Lane 3
3 ×××× 13.3 Gb/s Lanes
Decoder
PAM-4
CAP-2
PAM + CAP (Bi-phase )
t (ps)
200 300 4000 100
CAPSimple algorithm to separate 2 formats
– PAM: S(t) + S(t+∆τ)
– CAP: S(t) – S(t+∆τ)PAM t (ps)
PAM
CAP
f (GHz)
0 10 20-20 -10
Frequency spectra are partially overlapped
Band-pass filtering can be used
Data
– CAP: S(t) – S(t+∆τ)
∆τ = half period1 0 1 1
200 300 4000 100
Schematic of PAM -CAP demonstration
Data 1
Data 2
Clock
ττττ1
+
XOR
PAM
CAP
Encoder
TransmitterFiber
Decoder
Data 2Receiver
Data 1
Pow
er
CAPPAM
Band-pass filter
Low-pass filter
10GHz 10GHz
10GHz
FrequencyFrequency
Reasons for choosing PAM and CAP line codes
– Allows the use of low cost optical components and electronics
– Only requires a direct detection optical receiver prior to the decoder
– Reduces need for digital electronics for lane detection (complementary frequency distributions)
– Suitable for use with both single-mode and multi-mode optical fibre links
FrequencyFrequency
37.5 Gb/s Experimental Results – Eye Diagrams
Optical combined
Electrical combined CAP-2 and PAM-4
signal
Summary
• 12.5 Gb/s baud rate
• 37.5 Gb/s aggregated data
rate
80 ps
Optical combined CAP-2 and PAM-4 signal
Decoded PAM-4 signalQ = 3.7
Decoded CAP-2 signalQ = 4.8
rate
• 5-tap transversal filter for decoding
– 32 ps tap spacing
– 16 GHz bandwidth
• Optical signal SNR degradation comes from RF amplifier
noise
• Introduction to Short Link Systems
• Growing energy requirements in the internet
• Use of optics to reduce supercomputer/data center energy cost
• Modulation formats for next -generation optical
Key Themes
35
• Modulation formats for next -generation optical datacommunication links
• Candidate modulation formats
• Carrierless amplitude and phase modulation
• (If time) Reducing energy consumption of optical sw itches
• Active-passive OEICs
• Conclusions
Generic Integration Philosophy
Electronic integration
3 basic elements
Photonic integration
4 basic elements
WaveguidePWD
SOA PWD PHM
Phase
Amplitude
Polarisation
PWD
ϕ
ΑSOA
P
PHM
Photonic Integration with 4 basic Building Blocks
waveguide
curve
optical amplifier
λ converter, ultrafast switch
phase modulator
amplitude modulator
Passive Phase Amplitude
polarisation converter
pol. splitter / combiner
Polarisation
curve
AWG-demux
MMI-coupler
λ converter, ultrafast switch
picosecond pulse laser
multiwavelength laser
amplitude modulator
2x2 switch
WDM OXC
pol. splitter / combiner
pol. indep. 2x2 switch
pol. indep. diff. delay line
Integrated 16 x 16 Port Switch
b
ac
d
a) Beamsplitter
b) Gating SOAs
c) TIR mirrord
• Very compact footprint: 6.3 mm x 6.5 mm
• Contains 192 active switching elements
• Also an additional 922 features on the interconnecting shuffle networks
• Performance good, but limited by the long all-active paths – up to 16.8 mm
• All active, consumes 16 W (currently the shuffle networks consume ~60% of the drive current)
� Reduce power consumption by active-passive integration
d) Crossing
Active-Passive 4x4 SOA -Based Switch with Integrated Power Monitoring
1 x 2 MMI
power splitter
90 degree
Photodiode
2x2 MMI Power
splitter
Output passive
shuffle network
Booster SOA
Array
Input passive
shuffle network
Active Gating
SOA Array
90 degree
waveguide bend
• Active-passive integrated switch designed and fabricated using a generic design and foundry platform
• The switch is constructed from generic building blocks, such as passive waveguides, MMI (purple) and active gain blocks, photodiodes (white) etc.
• Monolithically integrated, high efficiency photodiode for power monitoring and control• Energy consumption reduce by 65% compared to all-active • 160Gb/s for 6W – 37pJ/bit (should scale to <10pJ/bit
IPDR Performance for Constant Current
• Fixed current of 26mA and 45mA applied to the gating and booster SOAs respectively
• The IPDR has been measured to be 15dB for a 2dB penalty (6dB for 1dB penalty) for the path 3-4 and a 20dB IPDR (10dB for 1dB penalty) for path 4-4
• Short haul data links are getting:• Faster• More power hungry• Harder to implement at low cost
• Modulation formats for next-generation datacommunic ation links
Conclusions
41
• Recent results in NRZ, PAM, QAM and OFDM reviewed
• Carrierless amplitude and phase modulation identifi ed as particularly suitable and initial demonstration per formed
• Active-passive integration for reduced energy photo nic circuit switches
