Benn ThomsenMicrosoft Research
Operating Optics at scale
Data Center
1 km radius
150,000 servers
120,000 100G optical transceivers
Regional Network
70km radius
512 fibre pairs
2Pb/s
WAN
Data centers in 50+ regions
More than 70,000km of fibre
Flops
Bytes
Operating vertically: The rack
Operating vertically: DC Network
Innovating vertically: Regional Architecture
15-20 data centers in a region
Mega Data Center
Distributed data centers across an 80Km region to keep pace
with cloud growth
5
Innovating in optics
Colored optics
WAN
Regional network
(80 km links)
Regional hub
Mega data center Regional architecture 6
Gray optics
» Dual-λ, 28 Gbaud PAM4
» < 4.5 W per module
» Variable gain/tilt EDFAs
» Automatic Tunable CD compensation
Regional architecture: PAM4 QSFP28 DWDM transceiver
EDFA
L2/3
sw
itch
36
xQSFP
Lin
eca
rd
Mux/
Dem
ux EDFA
TDCMx4
QSFP28
...
DCF
QSFP28QSFP28QSFP28QSFP28
QSFP28
QSFP28
QSFP28
fiber
span
VOA
x36
ADVA SmartAmp
ColorZ: Inphi 100G QSFP28
Optimised for 80km
Storage
Networking
Disruption across the board
Gb
ps/
$
WAN capacity
TimeG
B/$
Hard disk capacity
Time
Gb
ps/
$
Data center
network
Time
IOPS/$
Flash capacity
Time
FLO
PS/$
CPUs
Time
1980s
2006
Moore’s Law for NetworkingTb
ps
(lo
gsc
ale
)
Time
~2x every two years
$
Time
Cloud network cost
64
256
1024
4096
16384
2004 2006 2008 2010 2012 2014 2016 2018
Thro
ugh
pu
t (G
b/s
)
Year
Will free scaling continue?
12.8Tbps(Broadcom, Innovium)
256x 50Gb/s PAM4
2 bits/sym6.4Tbps
256x 25Gb/s NRZ
1bit/sym
Switches are becoming edge bandwidth limited
Moore’s Law for NetworkingTb
ps
(lo
gsc
ale
)
Time
~2x every two years
$
Time
Cloud network cost
11
Gb
ps/
$
Data center network
Opportunity for photonics (shorter-term)
PCB
Switch ASIC
Optics
Fiber PCB
ASIC Optics
Fiber
CMOS Scaling limit
PCB
Multiple ASICs
Fiber
Today 16nm CMOS
On board optics In package optics Optical Network on chip
In-package optics (IPO)
Chip
SwitchASIC
Tx/Rx
PCB
1
64
TX/RX
Tx/Rx
Tx/Rx
Transceivers
OpticalSource
• Challenges• Material: Silicon, InP• Approach: Direct modulation, MZM, Ring modulator, Coherent??• Real estate, yield, operating environment• Fibre attach
70mm
Opportunity for photonics (long-term)
Fundamental challenges− All optical header processing
− Optical buffers
✓More developed technology− Higher- radix enables a flat network
Optical circuit/burst switch
Optical packet switch
An old idea made newNew goalpost
Large Scaledata-center-wide
&
Fast switchingSub nano-second
Sca
le o
f so
lutio
n
Switching speed
Slow (ms) Fast (ns)
Chip
scale
Data center
scale
Optics community
Optical switch
Mechanically-moved
mirrors (MEMS)
Network community
Bridging the last mile
Apps
Network Stack
NIC
PHY
Apps
Network Stack
NIC
PHYOptical Circuit Switch
Needs to support Burst mode CDR
Network wide synchronisation
New applications that exploit the network properties e.g. low latency
We need to consider the implications for the whole system
Optical Network Prototype
Nanosecond-granularity switching
Long, long way to go!17
Integrated Photonics: 56 Gbaud Modulator Array
Output FibreAttach
4x 56Gbaud Modulators
2x 56G Ring Modulator
5mm
Chip testing
Storage
Hot
Online
Technology: SSD, HDD
$$$$$
Cool
Read Write Occasionally
Technology: HDD
$$$
Cold
Write Once, Read Once in a Blue Moon
Technology: Tape, Blu-ray, Optical?
$
Today’s cold storage
Credit: stock.adobe.com
Hard disk drive Tape
GB/$
Hard disk capacity
Writing in Silica glass
23
Voxel structure
• Approx. ellipsoidal shape, 𝛾~1𝜇𝑚, 𝑑~5𝑢𝑚
• Volumetric nanograting
• Exhibits form birefringence
• Multi-level encoding using 𝑹𝒆𝒕 and 𝜽 to store multiple bits per voxel
𝑑
𝒏𝟏 𝒏𝟐
Transverse view
𝜽
𝛾
Voxel Writing
Arrays of voxels
Information encoded by modulating the intensity and polarisation of the laser pulses
Challenge: Scaling write speed• Modulator bandwidth• Parallelism
Reading from Silica glass
Optical microscopy:Sample
Exploit developments in consumer camerase.g. Sony 16MPixel 1000fps
Voxel Reading Read demo
A big opportunity – cross-disciplinary approach
Optics
NetworkingStorage
Hardware
28
Optics for the Cloud
Physical
Network
Storage
New optics
New hardware/software
New Media
Application New scenarios
Custom PIC
Innovation across the cloud stackAt Microsoft and with collaborators
Prototype switch
Silica
Want to know more
www.opticsforthecloud.com