Next Wave of Data Centers & Interconnects

LIGHTCOUNTING Market Research on High-Speed Interconnects Datacom, Telecom, CATV, FTTX, Consumer markets © Copyright 2011 LightCounting LLC All material proprietary and confidential.

Next Wave of Data Centers & Interconnects

10GBase-T, DAC,

40G/100G transceivers,

AOCs, Embedded Modules

9/18/2012 1

Brad Smith Sr. VP LightCounting

[email protected]

Avago

mailto:[email protected]


Agenda

• What is going on in the data centers that is driving the interconnect space

• 10GBase-T, Direct Attach Copper

• 40G & 100G transceivers

• Active optical cables

• Embedded Optical modules

9/18/2012 2


About LightCounting

9/18/2012 3

http://www.lightcounting.com/reports.cfm

Transceiver Research Databases

High speed interconnect Research LH telecom & FTTx -to- Data centers Ethernet, Fibre Channel, InfiniBand Vertical areas: 40/100G, AOCs, EOMs





Problem with Optical Interconnects All Predictions are 35%-90% Traffic Growth

9/18/2012 4

IEEE BWA, Cisco VNI, LightCounting Model


Virtually all the Internet traffic – goes through an optical

transceiver somewhere in the telecom and datacom network!

9/18/2012 5

Virtually all the Internet traffic – goes through an optical

transceiver somewhere in the telecom and datacom network!

Supplied by transceiver companies… About one quarter away for bankruptcy


9/18/2012 6

Virtually all the Internet traffic – goes through an optical transceiver somewhere in the telecom and

datacom network!

Supplied by transceiver companies… About one quarter away from

bankruptcy!


Problem with Optical Interconnects What’s Wrong with this Picture?

9/18/2012 7

IEEE BWA, Cisco VNI, LightCounting Model Optical Stocks

Traffic going through roof Profits going through the floor


Big Changes in Data Centers Changes in Data Centers Drives Interconnects

9/18/2012 8


M&As Restructure the Industry ~24-months

Systems Cabling

Transceivers Semiconductors

9/18/2012 9


Cabling Companies Buy AOC Startups + BUs

• TE bought Zarlink BU • FCI Bought MergeOptics • Molex bought Luxtera BU • Samtec bought AlpenIO

Result: • $ xx Billion, international connector, cabling, companies • Offering transceiver and AOCs, EOM, VCSELs

– New set of competitors for transceiver companies – Full line of copper & optical products – Power cables, every protocol, every MSA, connector, etc

9/18/2012 10


Cisco Introduced Blade Servers Competes against former server partners

• HP bought 3Com/3Par

• IBM bought Blade Network Technologies

• Dell bought Force10

• Oracle bought Sun Microsystems

• Mellanox bought Voltaire (InfiniBand)

• Intel bought QLogic and Fulcrum (InfiniBand) & Crays HPC Intereconnect BU

Now integrated Server and Switching companies

9/18/2012 11


Cisco Buys Transceiver Companies CoreOptics & LightWire

• CoreOptics: 100G Long haul in-house

• LightWire: Silicon photonics in-house

– 4x25G (100G) -- WDM

– High speed modulation (40G) & integrated electronics

– Data center (500 meters) & Metro (10Km)

Result:

• Servers, switches, and interconnects – all in-house

9/18/2012 12


Software Defined Networking

• SDN Virtualizes the networking hardware

• VMWare bought Nicira

• Citrix bought Nutanix

• Oracle bought Xsigo

• Cisco Systems’ investment in Insieme

• Bottom line: – Big technology companies are buying their way into

the virtualized storage and virtualized networking markets.

– Virtualizations will likely increase all the DC traffic

9/18/2012 13


9/18/2012 14

Same thing that happened with PCs in 1980s with Taiwanese clone PCs and big name brands ……is happening to servers and switches! This will change everything!


Enter the White Box Clones

Servers

• Big Internet related companies (Google, Facebook, Amazon, etc.) buying servers directly from ODMs

• Making designs available for anyone to buy

• Server business already at 22% GM & heading lower

• NET: White box commodity servers!

Switches

• White box ToR and EoR switches

• Buy switch chips from Broadcom, Marvel, Mellanox, …Intel

• Run OpenFlow (open source network switching software)

• No longer special switching software play

• No longer a Microsoft-Intel instruction set lock

• NET: White box commodity switches!

9/18/2012 15


Reactions Reactions likely: • M&As: Buy transceiver companies Cisco/Lightwire • Advanced technology: Use technology as a weapon against

low cost manufacturing white boxes • More highly integrated Blade server-switch boxes • Rack-as-a-Sku” • Non-Intel Alternatives: ARM cores, Power7, SPARC

microprocessors competing with Intel • HPC technologies hit high-end data centers • Software Defined Networking

9/18/2012 16

“Rack as a Sku”

Integrated Blade Server/Switches

All these changes are going to rewrite the Server/Switch and Interconnect landscape going forward

Mez board Copper KR

Server blade

Switch blade


Server-Switch Architectures

9/18/2012 17

2 Sockets 4 Sockets

1RU Servers with Top-of-rack Switches Blade Servers with Integrated Switches

No single solution!


Server & Switch Industry Transitions

9/18/2012

Brad Smith [email protected]

18

“Rack as a Sku”

Traditional Rack Servers

& Top-of-rack switches DAC, QSFR SR & AOCs

SMF 10Km LR4 transceivers

Internally connected with EOMs

and AOCs

SMF 2Km nR4 transceivers

Multiple Blade Server Bays per Rack Custom (e.g Google)

& White Box Clones DAC, QSFR SR & AOCs

SMF 2Km nR4 transceivers

Today 2013-2014

2015+

From Major System Suppliers

Cisco, HP, IBM, etc.

Focus on advanced

architectures & long term

Focus on low prices &

3 yr life cycles


Waiting for New Servers Waiting for 10GBASE-T

Waiting for Europe’s economy to stop crashing

Preparing for the Exa Flood of Data

9/18/2012 19

State of the Industry Mid 2012


State of the Industry • Data centers are largely very cost sensitive;

• Cost governs all decision making (over technology) – Decision algorithm (Cost of new product) / Server cost

– Buy 100G LR4 transceiver OR 8 servers?

• Average and large data center needs 4-5G, not 10G yet

• Getting ready for 10G and 40G QSFP

• Most popular optical transceiver = 1G SFP+ & 10G SFP+

9/18/2012 20


State of the Industry

Servers & Switches

• Intel’s next gen microprocessor “Romley” server architecture – delayed ~6-9 mo’s

– 4 cores 10 cores (14 cores AMD) -- PCI Express Gen 2 Gen 3 with twice I/O

– QuickPath to bypass bus bottlenecks -- 32-nm very energy efficient

• Top-of-rack & End-of-row switches are 10G SFP+ + 40G QSFP ready but ….

• Entire industry was “Waiting for new Romley servers”

• And 10GBASE-T low-power 28-nm parts

9/18/2012 21


Intel’s Tick-Tock Model Sets the Clock for Data Center Upgrades

Tick= Architecture upgrade ; Tock= Silicon Shrink

9/18/2012 22

Data center spending delayed by 8 months


Microprocessors

9/18/2012 23

2-4 Cores 10-16 Cores 40-50 Cores

2 MPUs/Server 128GBytes DRAM PCIe Gen2 (8 lanes @5G/lane

? MPUs/Server 1T+ Bytes DRAM PCIe Gen3 (8G/lane) ARM, INTEL, AMD

4 MPUs/Server 768G Bytes DRAM PCIe Gen3 (8/16 lanes @8G/lane

? Data center in a server


I/O for Interconnects Virtualization software • Sends jobs to non busy MPUs

– Increases utilization from 20% to 90%

• I/O increases: Data Direct I/0 - transfer of data directly to from the network interface and cache, bypassing main memory

PCI Express • PCI Express Gen 2 5G 8b/10b and 8 lanes

• PCI Express Gen 3 8G 128/132 and 8-16 lanes

9/18/2012 24

PCIe Bus

20% 90%

Bottom line: ~200-%300 % faster I/O bandwidth


Servers per Rack

9/18/2012 25

48 - Severs / Rack 60-80 Severs / Rack


Traditional Data center Computer & Switch History Museum

9/18/2012 26


Modern Data center

9/18/2012 27

bestwebhostinggeek.com

keyitec.com


9/18/2012 28

.primecaretech.com

Mega Data Center


SuperComputer Site (HPC)

9/18/2012 29

18M Watts of power


Apple South Carolina Data Center

9/18/2012 30


Interconnect Technologies

9/18/2012 31


“Waiting for 10GBASE-T • Will be Reeaaaaly big!

• “When it gets small ….….. it will be big”

• Lan On Motherboard (LOM) is key to market

- PHY chip needs 28-nm (now at 40-nm)

- Late 2013/2014

- 10GBASE-T likely the last LOM implementation

• VCs spent $850M on 10+ startups

• Vendor community still shrinking

– Broadcom, Marvel are the 2 main players left

– Also Applied Micro, Aquantia in game

– PLX/Teranetics

• Sold Phy division to Aquantia

– Aquantia: Last remaining VC funded startup

9/18/2012 32

130-nm - This will be the year! 90-nm - This will be the year ! 65-nm - This will be the year ! 40-nm - This will be the year! 28-nm – This will be the year!

NIC

LOM

Main advantage: No huge markup like with Optical transceivers Will be big with LOM 2014


IEEE Update 40GBASE-T

• New CFI at the IEEE (July meeting) – everything is indeterminate at this point

• No vendor is anywhere close to payback 10GBASE-T investments so real support is likely to be slow – 2017-2018 event

• Designed for 2016 when Romley+ moves the server to 40G uplinks

• 85-90% use will be Server uplinks to top-of-rack switch

• 4 twisted pairs

• Encoding(?): 10G using differential signaling across pairs

• Scheme: 4x10G configuration

• Reach(?): 50 m (Not likely 100m anymore)

• Connector(?): Likely new RJ-4x; Likely not field terminated

• Cable(?): Cat8? Cat7a

– In any case – it’s a NEW cable! Meaning the “existing installed cable is not usable”

• Biggest advantage is = no optical price mark ups by suppliers

• Not likely to be supported by any startup so its Broadcom/Marvel game


Direct Attach copper • Has become very popular <5-7meters • Filling the 10GBASE-T void • Low-cost, low-power, Plug & Play • Cheaper than optical SFP+ • Active & Passive DAC • Outselling 10GBase-T 4:1

• Numerous problems surfacing

– Electrically linking different vendors equipment together directly – different than linking optically

– “Limiting” and “non-limiting” ports – no warning labels

• Net: “DAC in the Rack” • ---until 10GBASE-T LOM shows up

9/18/2012 34

10G = 7 meters 14G = 3meters 28G = ????


100G SR & LR4

• So called 2 Km solution really 4-6 dB issue

• IEEE is still working the technical issues on Reach 400m-1,200m

• 25G VCSELS maybe reach 25 meters!

– 125 meters with FEC chips added (more cost, more power)

• 50G maybe 12 meters without FEC

9/18/2012 35


Why 100G Needed in Data Centers 100G Driving Forces

• 48 – 80 Servers/ rack

• 10Gbps server uplinks

• 480G -800G problem!

• Will demand 100G uplinks

• Timing: Romley Q2/2012 : Volume 2H/2012

• ~ 2H/ 2013 need for 100G transceivers climb rapidly

• Then it pops to 40G server uplinks in 2015/16!

9/18/2012 36


100G

9/18/2012 37


9/18/2012 38

.primecaretech.com

Mega Data Center

Link this rack

To this rack

400-600metes Not 2Km! 4-6dB link budget


Active Optical Cables

9/18/2012 39


What is an Active Optical Cable? • Parallel optic transceiver in a standard MSA shell but… integrates

the fibers directly into the MSA

• No optical connectors

• Changes the game!

9/18/2012 40

Molex

Finisar


Active Optical Cables

• Very popular in High Performance computers (HPCs) TOP25

• Technology is proven out and is rippling down to Top 25-100 & High-end data centers

• Used to connect top-of-rack switches to end-of-row switches (4x10G QSFP)

• Large router companies are starting to use 12x10G CXP

• Advantages: – Plug & Play; green light;

– No optical connector cleaning, familiar copper connector;

– low power (700mW per end)

• InfiniBand today but moving to Ethernet, SAS, PCI Express

9/18/2012 41


1 Core AOC Technology Creates Multiple Products

• Multiple types – 1x10G SFP+ - 4x5G CX-4

– 4x10G QSFP - 4x14G QSFP

– 12x10G CXP - 12x12.5G CXP

– 8 x 8G iPass

• Breakout AOCs: 4-SFP/QSFP and 3-QSFP/CXP

• Embedded Optical Module (EOM) replacing SNAP12

– For inside chassis applications (no MSAs yet)

• Parallel Optical Transceivers

• AOC/Transceiver: AOC with connectorized fibers

• Multiple applications: HPC to consumer

9/18/2012 42

Avago

FCI

GigaLight

AlpenIO Photonera


9/18/2012 43

CES AOCs

LED lights To USB and HDMI cables To Supercomputers AOCs


Consumer Embedded AOCs 4x12.5G

9/18/2012 44


Embedded AOCs 4x12.5G

9/18/2012 45

Parallel

AOC


9/18/2012 46

Embedded Optical Modules The End of Pluggable Optics?


Embedded Optical Modules “Avago MicroPOD-Like Thingies”

• Inside chassis applications

• Mid-board optics

• Start out at 10G but the big market will be at 25G

• Start out in large routers; later move to servers

9/18/2012 47

Finisar

Avago

NEC


Three Generations of EOMs

9/18/2012 48


9/18/2012 49

16 MicroPods in 4x4 array

Try cleaning this optical mess!


9/18/2012 50

Its all about achieving high density interconnects


IBM HPCs & Avago MicroPODs

9/18/2012 51


9/18/2012 52

IBM


9/18/2012 53

IBM


9/18/2012 54

Massively complex mechanical assembly!


8-10 Companies Showing Prototypes 2-3 With Announced products

9/18/2012 55


Where Are EOMs Used

9/18/2012 56

Demo board from US Conec and Avago


Sony AOC/EOM CXN2006, CXN2007 and CXN2300 optical interconnection modules

• Designed for 4K x2K video for HDTVs and digital cameras

• 4x10G based on VCSELs with a lens-less internal design

• 10mm x 16mm

9/18/2012 57

http://www.sony.net/Products/SC- P/cx_news/vol68/pdf/featuring_vol68_interconnection.pdf

http://www.sony.net/Products/SC-




9/18/2012 58

Fujitsu


SNAP12 vs MiniPOD

9/18/2012 59

Clearly EOMs have an advantage over SNAP12s


NEC 12x25G EOM at OFC2011

9/18/2012 60


Where Are EOMs Used?

9/18/2012 61


9/18/2012 62

Where do you put an EOM?


9/18/2012 63

Avago EOMs


9/18/2012 64

Offers extremely high density Optical interconnects to board edge

125mW/ch @ 10G 12 channels simplex = 1.45W 8.8 W Duplex link would use 12 EOMs = 17W


Optically Enabled Hybrid Chip Packages

• FTTChip

– Mez cards

– Chip/Optical

9/18/2012 65

Reflex/Fulcrum


Embedded Optical Modules The End of Pluggable Optics?

• Answer=NO! But it is one more opportunity. – No standard electrical or optical connectors? – No interoperability requirements? – No standard configurations 4x, 12 channel? – Every supplier has unique products (no second source)

• The industry will be able to achieve enough density in the data center with pluggable optics for at least next 4-5 years – 8 CFP/2s running 100G = 800G – 36 QSFP28 running 100G = 3,600G

1. Significant concerns about mixing 70C optics in 100C ASICs 2. Biggest limiter is repairs on failure

– Pluggable = changing fan belt on car – EOM = changing the piston

3. Rigorous vendor qualification and selection 3. Few major players

9/18/2012 66

Biggest inhibitor of market adoption


Conclusion

• Data center are in tremendous change

– Impact or clone Servers/switches

– M&As re-writing the landscape

• Low cost AOCs and Embedded Optical Modules

• Transition to 25G signaling for SR and LR4

• 25G is likely to “thin the heard” in transceivers

9/18/2012 67


Thank You

9/18/2012 68

Our Website: www.lightcounting.com

Headquarters Location:

858 West Park Street Eugene, OR 97401 Tel: 408.962.4851 Email: [email protected]

[email protected]



Date post:	21-Nov-2014
Category:	Documents
Upload:	lightcounting
View:	14,152 times
Download:	4 times

Next Wave of Data Centers & Interconnects

Documents