OFC2016 OpticalInternetworkingForum 1
OIFCEI-56GApplicationNoteCommonElectricalInterfaceat56Gb/s
Abstract:TheOIFiswellalonginthedevelopmentoftheCEI-56GsuiteofhighspeedinterconnectImplementationAgreements(IA)s.TheOIFisdevelopingIAsfornine56Gclauses,spanningfivereachesandthreemodulationtechniques.
TheOIFhasalonglegacyofdevelopingIAsforserialelectricalinterfaces,withtheCEIfamilybeingtheprincipalindustrydefinitionforSerDesforthe6Gb/s,11Gb/sand28Gb/sgenerations.InterfacesthatleveragetheCEIfamilyofspecificationshavebeenbuiltinmostofthemajorASICflowsandFPGAdevices.CEIhasbeenadoptedoradaptedintomultipleinterfacesbyIEEE802.3,InfiniBandandFibreChannel.
AbouttheOIF:
TheOIFfacilitatesthedevelopmentanddeploymentofinteroperablenetworkingsolutionsandservices.MemberscollaboratetodriveImplementationAgreements(IAs)andinteroperabilitydemonstrationstoaccelerateandmaximizemarketadoptionofadvancedinternetworkingtechnologies.OIFworkappliestoopticalandelectricalinterconnects,opticalcomponentandnetworkprocessingtechnologies,andtonetworkcontrolandoperationsincludingsoftwaredefinednetworksandnetworkfunctionvirtualization.TheOIFactivelysupportsandextendstheworkofnationalandinternationalstandardsbodies.Launchedin1998,theOIFistheonlyindustrygroupunitingrepresentativesfromacrossthespectrumofnetworking,includingmanyoftheworld’sleadingserviceproviders,systemvendors,componentmanufacturers,softwareandtestingvendors.InformationontheOIFcanbefoundathttp://www.oiforum.com.
Foradditionalinformationcontact:
TheOpticalInternetworkingForum,48377FremontBlvd,Suite117,
Fremont,CA94538USA
OFC2016 OpticalInternetworkingForum 2
TableofContents
Abstract:.......................................................................................................................................................................1
AbouttheOIF:.......................................................................................................................................................1
Glossary†.....................................................................................................................................................................4
TheOIFCEILegacy..................................................................................................................................................7
Introduction................................................................................................................................................................8
MotivationforCEI-56G..........................................................................................................................................9
CEI-56GChallenges...............................................................................................................................................12
DieArea.................................................................................................................................................................12
ChipPowerDissipation..................................................................................................................................12
PackagePowerDissipation...........................................................................................................................12
ChipPowerWiring............................................................................................................................................12
SystemPowerDissipation–Mid-planes.................................................................................................12
SystemPowerDissipation–Mid-boardoptics.....................................................................................13
I/ODensitiesonChipsandConnectors...................................................................................................13
ChannelMaterialsandCharacteristics....................................................................................................13
ChannelReach....................................................................................................................................................14
Cables.....................................................................................................................................................................14
Latency...................................................................................................................................................................14
Summaryofchallenges...................................................................................................................................14
ModulationTechniques.......................................................................................................................................16
OFC2016 OpticalInternetworkingForum 3
NRZ..........................................................................................................................................................................16
ENRZ.......................................................................................................................................................................16
PAM-4.....................................................................................................................................................................17
InterconnectReachesandApplicationSpaces..........................................................................................19
USR-DietoDieInterconnectWithinAPackage.................................................................................20
USR-DietoOpticalEnginewithinaPackage.......................................................................................21
XSR-ChiptoNearbyOpticalEngine.........................................................................................................21
XSR-CPUtoCPU...............................................................................................................................................22
XSR–DSPArrays...............................................................................................................................................22
XSR-CPUtoMemoryStack..........................................................................................................................22
VSR-ChiptoModule.......................................................................................................................................22
MR-ChiptoChipwithinaPCBA................................................................................................................23
LR–ChiptoChipacrossaBackplane/Midplane.................................................................................23
LR–ChiptoChipacrossaCable.................................................................................................................24
InteroperabilityPoints........................................................................................................................................24
DieBump...............................................................................................................................................................24
PackageBall.........................................................................................................................................................24
ModuleInterconnect........................................................................................................................................24
Summary....................................................................................................................................................................25
TableofFigures
Figure1InterconnectApplicationSpaces....................................................................................................8
Figure2ScalingofGates,Bumps,PinsandI/O(Ref.Xilinx,I/OLineadded)............................10
Figure3InterconnectChallenges..................................................................................................................11
OFC2016 OpticalInternetworkingForum 4
Figure4TypicalNRZEyeDiagramandStatisticalEyePlot...............................................................16
Figure5ENRZTypicalStatisticalEyes........................................................................................................17
Figure6TypicalPAM-4Eyes...........................................................................................................................18
Figure7CEI-56GReaches.................................................................................................................................19
Figure8UseofCEI-56GReaches...................................................................................................................20
Figure9USR-DietoDie....................................................................................................................................20
Figure10USR-DietoOpticalEngine..........................................................................................................21
Figure11XSRChiptoOpticalEngine..........................................................................................................21
Figure13MR-ChiptoChip..............................................................................................................................23
Figure14LRChiptoChipacrossabackplane..........................................................................................23
Figure15PackageBumpInteroperabilityPoints...................................................................................24
Figure16ChiptoModuleInteroperabilityPoint....................................................................................25
Glossary†2.5D:Referstoatypeofdie-to-dieintegrationviaasiliconinterposerhavingthrough-siliconvias(TSVs)connectingitstopandbottommetallayers
3D:Referstoathree-dimensional(3D)integrateddeviceinwhichtwoormorelayersofactiveelectroniccomponents(e.g.,integratedcircuitdies)areintegratedverticallyintoasinglecircuitwherethrough-siliconvias(TSVs)arecommonlyusedfordie-to-dieconnection.
ApplicationSpaces:Portionsofequipmentornetworkarchitecturethatcouldbenefitfromhavingadefinedsetofinterconnectionparameters.
ASIC:Anapplication-specificintegratedcircuitisanintegratedcircuit(IC)customizedforaparticularuse,ratherthanintendedforgeneral-purposeuse.
BCH:Bose,Ray-Chaudhuri,Hocquenghemforwarderrorcorrection(FEC)codesareaclassofadvancedcyclicerror-correctingcodesthatareconstructedusingfinitefields.
OFC2016 OpticalInternetworkingForum 5
BER:BitErrorRatioisthenumberofbiterrorsdividedbythetotalnumberoftransferredbitsduringastudiedtimeinterval.
BGA:BallGridArray,apackagetype
CDR:Clockanddatarecovery,ablockinareceiverthatrecreatesaclockorclocksinareceiverbylookingatthestatisticaledgeinformationinthereceiveddataandthenusesthatclockorclockstosamplethereceiveddata.
CEI:CommonElectricalInterface,anOIFImplementationAgreementcontainingclausesdefiningelectricalinterfacespecifications.
ClockForwarding:AclockcanbeforwardedinparallelwiththedatatoallowatransceivertoavoidusingCDRinordertosavepower.ThisistypicallydoneinUSRapplications.
EMB:Effectivemodalbandwidth,seeTIA-492AAAD.
ENRZ:EnsembleNonReturntoZero,amulti-wirecodeinwhich3bitsaremodulatedwiththeHadamardTransformontofourwires.
FEC:Forwarderrorcorrectiongivesareceivertheabilitytocorrecterrorswithoutneedingareversechanneltorequestretransmissionofdata.
FPGA:FieldProgrammableGateArray–areprogrammablelogicdevicethatoftenincludesSerDes.
FR4:Agradedesignationassignedtoglass-reinforcedepoxyprintedcircuitboards(PCB).
Gb/s:Gigabitspersecond.Thestatedthroughputordatarateofaportorpieceofequipment.Gb/sis1x109bitspersecond.
GBd:Thebaudrateistheactualnumberofelectricaltransitionspersecond,alsocalledsymbolrate.OneGigaBaudis1x109symbolspersecond.
IA:ImplementationAgreements,whattheOIFnamestheirdefinedinterfacespecifications.
IC:IntegratedCircuit
I/O:InputOutput,acommonnamefordescribingaportorportsonequipment
ISI:Inter-SymbolInterference,theeyeclosurecausedbytheenergyremainingonthechannelfrompreviousunitintervals,whichtypicallyimpactsthehorizontaleyeopening.
ISI-Ratio:Theratioofthelargestcodeeyetothesmallestcodeeye,whichisasimplemetrictoevaluatetheimpactofISIonhorizontaleyeclosureforagivencode.
OFC2016 OpticalInternetworkingForum 6
LR:LongReach–longenoughtoreachachiplocatedacrossabackplane
MCM:Multichipmodule,aspecializedelectronicpackagewheremultipleintegratedcircuits(ICs),semiconductordiesorotherdiscretecomponentsarepackagedontoaunifyingsubstrate,facilitatingtheiruseasasinglecomponent(asthoughalargerIC).
Mid-boardoptics:anopticaltransceiverthatismountedonaPCBAawayfromthePCBAedge,closetoaswitchASICtoreducetheamountofPCBAtracelossbetweenanASICandtheopticaltransceiver.ThisisincontrasttothecommonpracticetodayoflocatingopticaltransceiversatthePCBAedge.
MUX/DEMUX:Multiplex/demultiplex,amultiplexer(ormux)isadevicethatselectsoneofseveralanalogordigitalinputsignalsandforwardstheselectedinputintoasingleline,Conversely,ademultiplexer(ordemux)isadevicetakingasingleinputsignalandselectingoneofmanydata-output-lines,whichisconnectedtothesingleinput.
MR:MediumReach–longenoughtoreachachiplocatedacrossaPCBAincludingonadaughter-card
NRZ:NonReturntoZero,abinarycodeinwhich1sarerepresentedbyonesignificantcondition(usuallyapositivevoltage)and0sarerepresentedbysomeothersignificantcondition(usuallyanegativevoltage),withnootherneutralorrestcondition.
PAM:Pulseamplitudemodulation,aformofsignalmodulationwherethemessageinformationisencodedintheamplitudeofaseriesofsignalpulses.
PAM-4:Pulseamplitudemodulation-4isatwo-bitmodulationthatwilltaketwobitsatatimeandwillmapthesignalamplitudetooneoffourpossiblelevels.
PCBA:Printedcircuitboard(PCB)assembly,anassemblyofelectricalcomponentsbuiltonarigidglass-reinforcedepoxybasedboard.
RS:ReedSolomonFECcoding,thisisatypeofblockcode.Blockcodesworkonfixed-sizeblocks(packets)ofbitsorsymbolsofpredeterminedsize.Itcandetectandcorrectmultiplerandomandbursterrors.
SerDes:Serializer/Deserializer
Tb/s:Terabitspersecond.Thestatedthroughputordatarateofaportorpieceofequipment.Tb/sis1x1012bitspersecond
USR:Ultra-ShortReach–justlongenoughtoreachanotherdiewithinthesamepackage
VSR:Very-ShortReach–longenoughtoreachatransceiverinafront-panelopticalmodule
OFC2016 OpticalInternetworkingForum 7
XSR:eXtra-ShortReach–longenoughtoreachanearbydeviceormid-boardopticalmodule
†Somedefinitionsincludecontentfromwww.wikipedia.com
TheOIFCEILegacyTheOIFhasalonglegacyofdevelopingIAsforhighspeedelectricalinterfaces.TheCEIfamilyhasbeentheprincipalindustrydefinitionforSerDesforthe6Gb/s,11Gb/sand28Gb/sgenerations.TheOIFSxIinterfacewasCEI’simmediatepredecessorandservedthe3Gb/sgeneration.TheOIFSPI/SFI3and4electricalinterfacesservedthe800Mb/sand1.6Gb/sgenerations.Thislonghistoryof7generationsandsixdoublingsgivestheOIFauniqueviewpointontheneedsoftheindustryfortodayandtomorrow.
InterfacesthatsupporttheCEIfamilyofspecificationshavebeenbuiltinmostofthemajorASICflowsandFPGAdevices.CEIhasbeenadopted,adapted,orinfluencedthedevelopmentofmultipleinterfacesincludingIEEE802.3,InfiniBandandFibreChannel,SATA,SAS,RapidIOandHyperTransportinterfacesaswellasnumerousproprietaryprotocols.
Name Rateperpair Year Adopted,AdaptedorInfluenced
CEI-56G 56Gb/s 2016 Thefutureisbright
CEI-28G 28Gb/s 2011 InfiniBandEDR,32GFC,SATA3.2,100GBASE-KR4and100GBASE-CR4,CAUI4,SAS-4
CEI-11G 11 2008 InfiniBandQDR,10GBASE-KR,10GFC,16GFC,SAS-3,RapidIOv3
CEI-6G 6 2004 4GFC,8GFC,InfiniBandDDR,SATA3.0,SAS-2,RapidIOv2,HyperTransport3.1
OFC2016 OpticalInternetworkingForum 8
SxI5 3.125 2002-3 Interlaken,FC2G,InfiniBandSDR,XAUI,10GBASE-KX4,10GBASE-CX4,SATA2.0,SAS-1,RapidIOv1
SPI4,SFI4 1.6 2001-2 SPI-4.2,HyperTransport1.03
SPI3,SFI3 0.800 2000 (fromPL3)
Introduction
Figure1InterconnectApplicationSpaces
www.oiforum.com 8
2 Introduction
2.1 Purpose The OIF Next Generation Interconnect Framework identifies application spaces for next generation systems and identifies areas for future work by the OIF and other standards bodies. The scope of this document explores Next Generation (NG) Interconnects that are limited to data center or intra-office applications which are generally less than 2km from both an electrical and optical perspective. Virtual Platforms in the cloud is an example of just one of the applications that will take advantage of NG Interconnect technology to achieve higher data bandwidths in a smaller footprint with better energy efficiency.
Figure 1 Interconnect Application Spaces
As shown in Figure 1, interconnection interfaces in a typical system are needed for chip-to-chip within a module, chip to chip within a PCBA (printed circuit board assembly), between two PCBAs over a backplane/midplane, or between two chassis’. These interfaces may be unidirectional or bi-directional, optical or electrical, and may support a range of data rates. For each application space, the IAs that follow from this framework should identify requirements to support interoperability across the various application spaces for optical and electrical links. They may include, but not be limited to:
� Cost Considerations � Link Performance � Power Consumption
OFC2016 OpticalInternetworkingForum 9
AsshowninFigure1,interconnectioninterfacesinatypicalsystemareneededforchip-to-chipwithinamodule,chiptochipwithinaPCBA(printedcircuitboardassembly),betweentwoPCBAsoverabackplane/midplane,orbetweentwochassis’.Theseinterfacesmaybeunidirectionalorbi-directional,opticalorelectrical,andmaysupportarangeofdatarates.
Considerationsfortheselinksmayinclude:
• Cost• LinkPerformance• PowerConsumption• Channellossbudgets• ChoiceofPCBmaterial• Modulationtechniqueandsignallevels• Numberoflanes,channelconfigurationandgeneralcharacteristics• Referenceclockjitter• ForwardedClockorCDR• Latency• Connectorperformance• Reliability• Size• OperatingTemperature
MotivationforCEI-56GNextgenerationsystemsarebeingdrivenbytheneedtohandletheincreasingvolumeofdatatraffic.Atthesametime,thenextgenerationsystemsareconstrainedbylimitsonpowerconsumption,bylimitsonthesizeofasystem,andbytheneedtoprovideacosteffectivesolution.
Theseneedsdrivenextgenerationsystemstoeverincreasingcommunicationportdensities.Theincreaseddensityleadstosmallersurfaceareasavailabletodissipatetheheatgeneratedandthereforerequiresdecreasedpowerconsumptionforaport.
Theindustrycurrentlyhaselectricalinterfacesfor10Gb/s(OIF’sCEI-11G),25Gb/s&28Gb/s(OIF’sCEI-25/28G)inproduction,andworkiscomingalongon56Gb/s(OIF’sCEI-56G).However,channelsproducedwithcopperPCBtracesareseverelybandwidthlimited,andbecauseofthisitisincreasinglydifficulttoachievethesamelinkdistancesusinghighersignalingrates.
ImprovementsinICintegration,whichhasbeenrelentlesslydrivenbyMoore’sLawoverpastdecades,haveenabledhigherdensitiesoflogicgatesatescalatinghigherclockratesto
OFC2016 OpticalInternetworkingForum 10
beusedinICdesigns.Thesetrendshaveallowedtheindustrytodeployincreasinglymorecomplexcommunicationsystemsateachgenerationtomeettheinfrastructureneeds.
However,asonedigsdeeper,thefutureappearstobechallenging.Manydifferenttechnologiesneedtoconvergetoimprovethethroughput.ThesecomplexICshaveincreasinggatecounts,butthepowerdissipationpergateandI/Ospeedhavenolongerscaledatthesamerateasthegatecount.Inaddition,thenumbersofelectricalconnections(bumpsandpackagepins)arealsonotscalingatthesamerate-leadingtoapower,capacityandportcountgapforthenextgenerationinterconnectinterfaces.
Figure2ScalingofGates,Bumps,PinsandI/O(Ref.Xilinx,I/OLineadded)
Thepredominantinterconnectchallengestoovercomearepresentedbelowinasolutionspacediagram,andarediscussedingreaterdetailinsubsequentsub-sections.
OFC2016 OpticalInternetworkingForum 11
Figure3InterconnectChallenges
OFC2016 OpticalInternetworkingForum 12
CEI-56GChallenges
DieArea
SerDesimplementingCEI-56GareexpectedtobelargerthanSerDessupportingCEI-28Ginterfacesbecauseoftheneedformoreadvancedequalization.Additionallythesizeoftheanalogportionoftheinterfacesdonotscaleaswellasthedigitalportions.
Thisareasituationhasrenderedsomepreviouschiparchitecturesunuseable.ThishasledtoatrendofSerDes“out-boarding”whereanUltra-ShortReach(USR)oreXtraShortReach(XSR)interfaceisputonthemainASICandthelongerreachSerDesareputinthesamepackageornearby.
ChipPowerDissipation
SerDesimplementingCEI-56GareexpectedtobehigherpowerthanSerDessupportingCEI-28Ginterfacesbecauseoftheneedformoreadvancedequalization.Additionallythepowerdissipationoftheanalogportionoftheinterfacesdonotscaleaswellasthedigitalportions.
ThishasfurtheredthetrendmentionedaboveofSerDes“out-boarding”whereanUltra-ShortReach(USR)interfaceisputonthemainASICandthelongerreachSerDesareputinthesamepackage.
PackagePowerDissipation
Whenpackagepoweristhekeyconstraint,theuseofanXSRinterfacecanhelpmovesomeofthepowerdissipationoutsideofthemainASICandontootherchipsnearbyonthePCBA.
ChipPowerWiring
Lowervoltagepowersourcesareusedwithsmallertechnologynodes.Theneteffectofhigherintegrationoflowvoltagesemiconductorsisasignificantincreaseintotaldevicecurrent,whichrequireshighcurrentsourcepowersupplieswhichmustbecontrolledwithinseveralmVtolerances,whichinturnrequiresadditionalpowerpinsperdevicetoaccommodatethehighelectricalcurrents.
SystemPowerDissipation–Mid-planes
Becausesystemcoolingcanescalatebeyondphysicallimits,multiplepowersavingstrategiesmaybeneeded.
OFC2016 OpticalInternetworkingForum 13
SincetheI/Opowerisrelatedtothedistancethattheelectricalsignalstravelforagivenchannel’sproperties,reducingthedistancethattheelectricalsignalmustbedrivencanreducepowerdissipation.Onetechniquethathasbeenusedistousemid-planes.
Withthisphysicalarchitecture,aswitchboardismountedhorizontallybehindtheverticallymountedfrontboards.Withthisarchitecture,theelectricallinksareshorterandonlyhavetotraverseoneconnector.
SystemPowerDissipation–Mid-boardoptics
Insomecases,itmaymakesensetointegratetheopticsclosetotheASICpackage,therebyminimizingtheneedtodriveelectricalsignalsfar.Thesearesometimescalledopticalenginesandthetechniqueisreferredtoasusingmid-boardoptics.
I/ODensitiesonChipsandConnectors
ThemaximumnumberofusefulI/Osforhighspeedseriallinksperdeviceisnotonlylimitedbytheavailablepackagetechnologyitself,butalsobytheabilitytoroutethedeviceonthePCBA.
Inordertomaintainsignalintegrityforahighspeedseriallinkdesign,itisrequiredtobeabletorouteadifferentialpairbetweentwopackageballswhenescapingfromtheinnerballrowsofaballgridarray(BGA),anditthereforemaybecomemorecostlytousepackageswithaballpitchbelow1.0mm.
Inaddition,foreveryballrowfromtheedgeofthepackageonwhichdifferentialpairshavebeenplaced,aseparatecircuitpackagelayerhastobeused.Differentialpairsintheouter4rowsoftheBGArequires4signallayersonthePCBA,whilethe6outerrowswouldrequire6layers,andthusthePCBlayerstackgrowswitheveryinnerBGArowtoberouted.
ChannelMaterialsandCharacteristics
Linkapplicationsarecharacterizedbythesupportedlossbudgetandsignalimpairments.Lossisdeterminedbylinklength,boardmaterials,back-drilling,viatypeusage,thenumberofconnectors,connectortypes,andpackagematerials.Increasesinsignalingratecausemorefrequency-dependentlossandthusalowerSignaltoNoiseRatio(SNR).
Thedeviationofthelossfromasmoothcurveisalsoimportantastheripplesinalosscurvearenoteasilyequalizable.ThisiscalledInsertionLossDeviation(ILD).
Ingreenfieldapplications,advancedmaterialsforthechannel(PCBandconnector)canresultinandecreasedlossandcanthussupporteitheranincreasedelectricaldatarateoradecreasedamountofrequiredequalization.
OFC2016 OpticalInternetworkingForum 14
ChannelReach
Incommunicationsystemsusuallythefrontplateareaisoccupiedbypluggablemodulesforinter-systemcommunications,whiletheintra-systemtrafficbetweenthePCBAsisconnectedoverthebackplane/midplane.Tosupportreasonablesystemdimensions,systembackplane/midplaneconnectionstypicallyneedtobridgedistancesofupto70–100cm.Insomemorerecentapplications,alternativechannelarchitecturesreplacethebackplanewithdirectplugorthogonal(DPO)structureswhereI/Olinecardsonthefrontsideoftheequipmentdirectlyplugintoswitchfabriclinecardsontherearsideoftheequipment,thusdirectlyconnectingthelinecardswithoutthelossofthebackplanebetweenthem
Theintroductionofrepeaterdevicesintothedatapathcanincreasetheeffectivereachattheexpenseofpowerandboardarea.
Cables
Twinax,micro-coaxand/orflexcircuitsarealsoanoptiontoreducelossandextendreachforthehighspeedlinks,particularlyinsystemswithfewerhighspeedlinks.Insomecircumstances,thiscanallowthemainPCBAtobebuiltfromlowcostmaterial.InanothercasecableassembliesarenowbeingintegratedontobackplaneconnectorandusedinlieuofPCBbackplanesduetotheirabilitytoreducelossandextendreach.Theuseofcablesbringstheirownshareofchallenges.
Latency
CPU-to-CPUandCPU-to-MemoryapplicationsofCEIareoftenintolerantoflatency.ThisoftenprecludestheuseofheavyweightFECintheseapplications.Thecharacteristicmethodthatisintolerantoflatencyistheuseofcredit-basedflowcontrolwhereareceivergrantsthetransmittertherighttosendacertainamountofdata.Onlyamoderateamountofbufferingistypicallyprovidedinthesesystems,mostofteninmorecostlySRAM.MostofthemajorCPUtoCPUprotocolsusecredit-basedflowcontrol.
Summaryofchallenges
Astimeproceeds,ICswillbecomefasteranddenser.Tocopewiththeissueofinterconnectcapacityanddensityoffuturesystems,photonicinterconnectswillbecomeanevenmoreimportantconnectiontechnology.
Theimplementationof56Gb/sInterconnectstechnologyposesseveralchallengesespeciallyinrelationto:diearea,chippowerdissipation,packagepowerdissipation,systempowerdissipation,limitedI/Odensity,channelloss,channelreach,andlatency.Highlightedweretheside-effectsofsomesolutions,whicharearesultofthecomplexinter-
OFC2016 OpticalInternetworkingForum 15
dependenciesof:higherintegration,complexmodulationschemes,chipbreak-outandrouting,signalconditioning,thermal&powerissues,andpackagefootprint.
OFC2016 OpticalInternetworkingForum 16
ModulationTechniquesAdvancedmodulationtechniquescanbeusedtolowertheNyquistfrequencyrequirementforalinkusingabandwidthlimitedelectricalchannel.ThesetechniquesrequireahigherSNRforequivalentBER.Theprovisionofchannelswithmorecorrelationstomeasureagainstcanalsohelp.Forwarderrorcorrection(FEC)canincreasetheeffectiveSNRandthereforethesupportedlossbudgetattheexpenseofhigherpowerdissipation,complexity,andlatency.Bandwidthlimitationsandincreasesinsignalingrateresultinimpairmentsthatmaybecompensatedbyusingequalizationtechniques,whichthemselvesalsoimpactpowerconsumptionandcomplexity.
NRZ
NRZ(NonReturntoZero)isthemodulationtechniqueusedbymostcurrentspeedlinks.Itrequiresachannelwithtwocorrelatedconductors.Itusestwosignallevelsandconveysonebitperbaud.Tosupport56Gb/sperpair,NRZrunsatasymbolrateof56GBaudandhasaNyquistfrequencyof28GHz.IthasanISI-Ratioofone,meaningtheratioofthelargesteyetothesmallesteyeisalwaysthesame.
At56Gb/s,NRZisusefulinapplicationswherethechannel’sfrequencydependentlossisnotexcessive.Thatlosshasbeenfoundtobeexcessiveinmanylongreachapplications.
Figure4TypicalNRZEyeDiagramandStatisticalEyePlot
ENRZ
EnsembleNRZ(ENRZ)requiresachannelwithfourcorrelatedconductors.Usefulchannelscanbeconstructedwithtwocorrelatedlooselycoupledpairs.Itusesfoursignallevelsandconveysthreebitsperbaudoverthefourconductorsbymodulatingthethreesub-channelscollectivelyusingtheHadamardmatrix.ItTosupporta56Gb/sperpaireffectiverate,ENRZrunsatasymbolrateof
�������� �������� ��������������������������������� � � � � �
Figure 9 shows the LR SERDES implemented in adiscrete retiming device. with b At the 56G data rate thereare proposals for PAM4 and ENRZ modulation.
�+)52( �
�+)52(
NRZ vs PAM4
The OIF made the decision to specify modulationformats of both NRZ and PAM4 for a number of reaches.This decision was based on the lack of clear technicaldata showing that either was a superior solution. Thiswhite paper addresses only a few of the comparisonsbetween NRZ and PAM4. A more detailed analysis is leftto the proponents of each modulation scheme.
Why NRZ?
NRZ has been the proven modulation technique forprinted circuit board electrical interconnects since thebeginning of digital communications. As data rates haveincreased, the use of first linear equalization and thendecision feedback equalization have kept NRZ as a viabletechnology. At both 10G and 25G data rates specificationshave been written for PAM4 based LR/backplane solutionsbut NRZ has been the clear market winner. Figure 10shows a 56G NRZ eye diagram.
�+)52( �� �����#�(9(�'+$)2$.
The use of two level signaling (0 and 1) provides a 9dBSNR advantage over PAM4 (4 levels). This provides ageneral rule of thumb that the insertion loss of a givenchannel measured at the PAM4 nyquist frequency shouldbe 9dB less than the insertion loss at the NRZ Nyquistfrequency (2x the PAM4 Nyquist frequency) before PAM4should be considered. Using this rule of thumb on the 28GVSR insertion loss curves extended to 56G (See Figure X)shows NRZ to be the best choice.
Why PAM4?
As Common Electrical Interfaces have increased indata rates system vendors have always requested supportto the same printed circuit board trace distances. This hasresulted in the use of lower loss PCB materials andimprovements in equalization techniques. At some pointthe additional ’knob’ of higher order modulation needs tobe used to achieve the same trace distance with higherdata rates. PAM4 modulation uses 4 voltage levels toencode 2 bit of data in single baud period. This result in aNyquist frequency that is 1/2 that required for NRZ. FigureZ shows a PAM4 eye diagram.
�������� �������� ��������������������������������� � � � � �
Figure 9 shows the LR SERDES implemented in adiscrete retiming device. with b At the 56G data rate thereare proposals for PAM4 and ENRZ modulation.
�+)52( �
�+)52(
NRZ vs PAM4
The OIF made the decision to specify modulationformats of both NRZ and PAM4 for a number of reaches.This decision was based on the lack of clear technicaldata showing that either was a superior solution. Thiswhite paper addresses only a few of the comparisonsbetween NRZ and PAM4. A more detailed analysis is leftto the proponents of each modulation scheme.
Why NRZ?
NRZ has been the proven modulation technique forprinted circuit board electrical interconnects since thebeginning of digital communications. As data rates haveincreased, the use of first linear equalization and thendecision feedback equalization have kept NRZ as a viabletechnology. At both 10G and 25G data rates specificationshave been written for PAM4 based LR/backplane solutionsbut NRZ has been the clear market winner. Figure 10shows a 56G NRZ eye diagram.
�+)52( �� �����#�(9(�'+$)2$.
The use of two level signaling (0 and 1) provides a 9dBSNR advantage over PAM4 (4 levels). This provides ageneral rule of thumb that the insertion loss of a givenchannel measured at the PAM4 nyquist frequency shouldbe 9dB less than the insertion loss at the NRZ Nyquistfrequency (2x the PAM4 Nyquist frequency) before PAM4should be considered. Using this rule of thumb on the 28GVSR insertion loss curves extended to 56G (See Figure X)shows NRZ to be the best choice.
Why PAM4?
As Common Electrical Interfaces have increased indata rates system vendors have always requested supportto the same printed circuit board trace distances. This hasresulted in the use of lower loss PCB materials andimprovements in equalization techniques. At some pointthe additional ’knob’ of higher order modulation needs tobe used to achieve the same trace distance with higherdata rates. PAM4 modulation uses 4 voltage levels toencode 2 bit of data in single baud period. This result in aNyquist frequency that is 1/2 that required for NRZ. FigureZ shows a PAM4 eye diagram.
�������� �������� ��������������������������������� � � � � �
Figure 9 shows the LR SERDES implemented in adiscrete retiming device. with b At the 56G data rate thereare proposals for PAM4 and ENRZ modulation.
�+)52( �
�+)52(
NRZ vs PAM4
The OIF made the decision to specify modulationformats of both NRZ and PAM4 for a number of reaches.This decision was based on the lack of clear technicaldata showing that either was a superior solution. Thiswhite paper addresses only a few of the comparisonsbetween NRZ and PAM4. A more detailed analysis is leftto the proponents of each modulation scheme.
Why NRZ?
NRZ has been the proven modulation technique forprinted circuit board electrical interconnects since thebeginning of digital communications. As data rates haveincreased, the use of first linear equalization and thendecision feedback equalization have kept NRZ as a viabletechnology. At both 10G and 25G data rates specificationshave been written for PAM4 based LR/backplane solutionsbut NRZ has been the clear market winner. Figure 10shows a 56G NRZ eye diagram.
�+)52( �� �����#�(9(�'+$)2$.
The use of two level signaling (0 and 1) provides a 9dBSNR advantage over PAM4 (4 levels). This provides ageneral rule of thumb that the insertion loss of a givenchannel measured at the PAM4 nyquist frequency shouldbe 9dB less than the insertion loss at the NRZ Nyquistfrequency (2x the PAM4 Nyquist frequency) before PAM4should be considered. Using this rule of thumb on the 28GVSR insertion loss curves extended to 56G (See Figure X)shows NRZ to be the best choice.
Why PAM4?
As Common Electrical Interfaces have increased indata rates system vendors have always requested supportto the same printed circuit board trace distances. This hasresulted in the use of lower loss PCB materials andimprovements in equalization techniques. At some pointthe additional ’knob’ of higher order modulation needs tobe used to achieve the same trace distance with higherdata rates. PAM4 modulation uses 4 voltage levels toencode 2 bit of data in single baud period. This result in aNyquist frequency that is 1/2 that required for NRZ. FigureZ shows a PAM4 eye diagram.
OFC2016 OpticalInternetworkingForum 17
37.3GBaudandhasaNyquistfrequencyof18.6GHz.IthasanISI-Ratioofone,meaningtheratioofthelargesteyetothesmallesteyeisalwaysthesame.ENRZcanoftenbeusedwithoutForwardErrorCorrection,evenonsomelongreachchannels.
Ata56Gb/seffectiverate,ENRZisusefulinlongerreachapplications,particularlywhentheuseofFECisnotdesired.
Figure5ENRZTypicalStatisticalEyes
PAM-4
PAM-4requiresachannelwithtwocorrelatedconductors.Itusesfoursignallevelsandconveystwobitsperbaud.Tosupport56Gb/sperpair,PAM-4runsatasymbolrateof28GBaudandhasaNyquistfrequencyof14GHz.IthasanIntersymbolInterferenceRatio(ISI-Ratio)ofthree,meaningthelargestcodeeyeisthreetimesaslargeasthesmallestcodeeye.Ithasaverticalimpairmentof9dBascomparedtoNRZandhasahorizontalimpairmentduetoitshighISI-Ratio.PAM-4isusuallypairedwithequalizationandForwardErrorCorrection.
At56Gb/sPAM-4isusefulinapplicationswherethechannel’sfrequencydependentlosswouldotherwisebeexcessive.
25
25
• The chart below graphs the horizontal opening vs. modulation technique over a TE Connectivity Channel with equal throughput per-wire
• PAM-4 (red) and NRZ (orange) are very similar, ENRZ (purple) is consistently better
Horizontal eye opening analysis Channel for comparison
Source: TE Connectivity
Differential
ENRZPAM-4
Comparison Eye Diagrams
PAM-4 @ 14 GHz NyquistNRZ @ 28 GHz Nyquist ENRZ @ 18.66 GHz Nyquist
ISI-ratio = 1!but frequency high
ISI-ratio = 3!bad ISI performance
ISI-ratio = 1!reasonable frequency!balanced eye shape
Same scale for each!112 Gb/s throughput over 4 wires for each
OFC2016 OpticalInternetworkingForum 18
Figure6TypicalPAM-4Eyes
�������� �������� ��������������������������������� � � � � �
�+)52( �������(9(�'+$)2$.
PAM modulation has been used for many years as thesolution of higher data rates over Category 3 and 5 twistedpair wiring for Ethernet. Figure Y shows an example of a28G VSR compliant channel that would benefit from theuse of PAM4 modulation to double the data rate with nochanges in PCB materials. Note that the loss difference atthe NRZ Nyquist frequency vs the PAM4 Nyquistfrequency is well beyond 9dB.
�+)52( ��������&*$//(-
The CEI 56G Technology ShowcaseIn an effort to highlight the progress of the OIF in the
CEI 56G projects a technology showcase was developedwith the intent of demonstrating technical feasibility of thedifferent reaches. There are 6 demonstrations fromvarious members of the OIF.
Demo 1: CEI-56G VSR NRZ QSFP channel demoThis demonstration shows a working VSR channel
using Test equipment from Tektronix to generate a50Gbps NRZ signal which is sent thru commerciallyavailableQSFP connectors on test boards from Multilaneand Yamaichi. (See Figure ?)The receiver is acommercially available NRZ receiver from Credo. TheBER of the system is shown on a PC using the BER testerin the Credo receiver.
�+)52( �����(.0��
Demo 2: CEI-56G-VSR PAM4 QSFP Complianceboard
The ability to transmit and receive PAM4 signals acrossa commercially available connectors is shown in Figure 14using test equipment from Tektronix, Multilane andAnritsu. One demonstration uses a Tektronix PAM4 signalgenerator to transmit PAM4 formatted data over a QSFPconnector using test boards from Molex and TEConnectivity. The PAM4 receiver is provided by Multilane.An additional demonstration uses an Anritsu PAM4 datagenerator sending signals through a CFP connector usingtest boards from Yamaichi and Multilane. The PAM4receiver is provided by Tektronix.
�������� �������� ��������������������������������� � � � � �
�+)52( �������(9(�'+$)2$.
PAM modulation has been used for many years as thesolution of higher data rates over Category 3 and 5 twistedpair wiring for Ethernet. Figure Y shows an example of a28G VSR compliant channel that would benefit from theuse of PAM4 modulation to double the data rate with nochanges in PCB materials. Note that the loss difference atthe NRZ Nyquist frequency vs the PAM4 Nyquistfrequency is well beyond 9dB.
�+)52( ��������&*$//(-
The CEI 56G Technology ShowcaseIn an effort to highlight the progress of the OIF in the
CEI 56G projects a technology showcase was developedwith the intent of demonstrating technical feasibility of thedifferent reaches. There are 6 demonstrations fromvarious members of the OIF.
Demo 1: CEI-56G VSR NRZ QSFP channel demoThis demonstration shows a working VSR channel
using Test equipment from Tektronix to generate a50Gbps NRZ signal which is sent thru commerciallyavailableQSFP connectors on test boards from Multilaneand Yamaichi. (See Figure ?)The receiver is acommercially available NRZ receiver from Credo. TheBER of the system is shown on a PC using the BER testerin the Credo receiver.
�+)52( �����(.0��
Demo 2: CEI-56G-VSR PAM4 QSFP Complianceboard
The ability to transmit and receive PAM4 signals acrossa commercially available connectors is shown in Figure 14using test equipment from Tektronix, Multilane andAnritsu. One demonstration uses a Tektronix PAM4 signalgenerator to transmit PAM4 formatted data over a QSFPconnector using test boards from Molex and TEConnectivity. The PAM4 receiver is provided by Multilane.An additional demonstration uses an Anritsu PAM4 datagenerator sending signals through a CFP connector usingtest boards from Yamaichi and Multilane. The PAM4receiver is provided by Tektronix.
�������� �������� ��������������������������������� � � � � �
�+)52( �������(9(�'+$)2$.
PAM modulation has been used for many years as thesolution of higher data rates over Category 3 and 5 twistedpair wiring for Ethernet. Figure Y shows an example of a28G VSR compliant channel that would benefit from theuse of PAM4 modulation to double the data rate with nochanges in PCB materials. Note that the loss difference atthe NRZ Nyquist frequency vs the PAM4 Nyquistfrequency is well beyond 9dB.
�+)52( ��������&*$//(-
The CEI 56G Technology ShowcaseIn an effort to highlight the progress of the OIF in the
CEI 56G projects a technology showcase was developedwith the intent of demonstrating technical feasibility of thedifferent reaches. There are 6 demonstrations fromvarious members of the OIF.
Demo 1: CEI-56G VSR NRZ QSFP channel demoThis demonstration shows a working VSR channel
using Test equipment from Tektronix to generate a50Gbps NRZ signal which is sent thru commerciallyavailableQSFP connectors on test boards from Multilaneand Yamaichi. (See Figure ?)The receiver is acommercially available NRZ receiver from Credo. TheBER of the system is shown on a PC using the BER testerin the Credo receiver.
�+)52( �����(.0��
Demo 2: CEI-56G-VSR PAM4 QSFP Complianceboard
The ability to transmit and receive PAM4 signals acrossa commercially available connectors is shown in Figure 14using test equipment from Tektronix, Multilane andAnritsu. One demonstration uses a Tektronix PAM4 signalgenerator to transmit PAM4 formatted data over a QSFPconnector using test boards from Molex and TEConnectivity. The PAM4 receiver is provided by Multilane.An additional demonstration uses an Anritsu PAM4 datagenerator sending signals through a CFP connector usingtest boards from Yamaichi and Multilane. The PAM4receiver is provided by Tektronix.
OFC2016 OpticalInternetworkingForum 19
InterconnectReachesandApplicationSpacesThefiveCEI-56Greachescanbesummarizedinthefollowingtable:
Figure7CEI-56GReaches
Theuseofthesereachesforinterconnectapplicationspacescanbebrokendownasfollows.
OFC2016 OpticalInternetworkingForum 20
Figure8UseofCEI-56GReaches
USR-DietoDieInterconnectWithinAPackage
Figure9USR-DietoDie
Itmaybenecessarytousemultipledieswithinamulti-chipmodule(MCM)toachievethepowerobjectives.Theseco-packagedsolutionscancommunicatewithlesspowersincethesubstrateprovidesahighqualitycommunicationchannel.
Thecommunicationchannelislessthan10mm.Thisshortelectricallinkmayallowforamuchsimplerinterfaceandrequirelesspowerthananexistingstandardelectricalinterface.Forexample,equalizationisunlikelytobeneededanditmaybepossibletoassumesuchshortlinksaresynchronous(singlereferenceclockgoingtoallchips),removingtheneedforafrequencytrackingCDR.
���� ����-2%0&!#%1�&.0���
�
��
�,�)�����'�
���
���
��
������
OFC2016 OpticalInternetworkingForum 21
Atypicaluseforthisistooff-boardtheSerDesfromaswitchASIC.
ClockforwardingcanbeusedintheplaceofCDRforUSRlinks.
USR-DietoOpticalEnginewithinaPackage
Figure10USR-DietoOpticalEngine
Itmaybenecessarytouseadieandanopticalenginewithinamulti-chipmodule(MCM)toachievetheindustry’sobjectives.Theseco-packagedsolutionscancommunicatewithlowpowersincethesubstrateprovidesahighqualitycommunicationchannel.
Thecommunicationchannelwouldtypicallybelessthan10mm.Thisshortelectricallinkmayallowforamuchsimplerinterfaceandrequirelesspowerthananexistingstandardelectricalinterface.
XSR-ChiptoNearbyOpticalEngine
Figure11XSRChiptoOpticalEngine
Itmaybeusefultoplaceanopticalinterfaceveryclosetothehostchip.SomeopticaldevicescannotsitwithinahostMCMduetoheatrestrictionsoftheopticalcomponents.Inthiscase,ashortelectricallinkoflessthan50mmisanticipated.Theshortreachofthischannelallowspowertobesaved.Theseopticalmodulesareoftencalledmid-boardoptics.
OFC2016 OpticalInternetworkingForum 22
XSR-CPUtoCPU
CPUscanbeconnectedviaashortconnectionoverhighrateSerDes.SomeCPUsuseacoherencyprotocoltomakethedifferentprocessorsappeartobeinthesamecoherencydomain.
Itisgenerallynotpossibletouseheavy-weightForwardErrorCorrectioninCPUapplicationsduetothelatencyrequirementsoftheseinterfaces,whichtypicallyusecredit-basedflowcontrol.
XSR–DSPArrays
DSPsaresometimesconnectedinarraystoprocesshighrateinformationsuchasfromaRADARoraLIDAR.Anautonomousvehicleisanexampleofsuchanapplication.Sometimestheinterfacesareunidirectionalandothertimes,bidirectional.BotharetypicallylatencysensitivesimilartoCPUsaseventheunidirectionalapplicationsoftenhavecomplex,timesensitivecontrolflows.
XSR-CPUtoMemoryStack
CPUscanbeconnectedviaashortconnectiontoamemorystack.Aclassicwaytoaccomplishthisistohaveadevicemadewithalogicprocessthatformsthebaselayerdevicefortheactuallymemorydies.
Itisnotpossibletouseheavy-weightForwardErrorCorrectioninthisapplicationduetothelatencyrequirementsofmemoryinterfaces.CPUthreadsorstatemachinesareoftenwaitingfortheresultsofthememoryaccess.Memorycacheshelp,butdonoteliminatethesignificantperformancehitsthatCPUsseewhenlatencyisadded.
VSR-ChiptoModule
Itiscommoninmoderncommunicationsystemstosupportpluggablemodulesatthefrontfaceplateoftheequipment.Thisfacilitateslowcostinitialdeploymentoftheequipmentifsomeportsareleftunpopulated.Apayasyougopolicyisthenuseduntiltheentirefaceplateispopulatedwithpluggablemodules.Theelectricallinkusedtoconnectthesepluggablemodulescanextendtobeyond30cm.Athigherdataratesthischallengestheabilityofthehostchiptodrivetheselongtracelengthswithinthepowerconstraintsoflargeswitchchips.Placingretimingdevicesinsidethepluggablemoduleprovidessupportforlongerhosttracesbuttheinclusionofcomplexequalizationfeaturescanoverburdenthelimitedpowerbudgetsofthepluggablemodule.Advancedmodulationformats(suchasPAMorDMTschemes),ForwardErrorCorrection(FEC)andequalizationfeaturesareallpossiblesolutionsforthechiptomoduleinterconnect.
OFC2016 OpticalInternetworkingForum 23
OnetechniquethathasbeenusedistoengineertheelectricallinkstohavealowernativeBERthanthatoftheopticallink.Inthisway,asingleFECcanbeusedtoprotectaCEI-56Glinkateachendandanopticallinkinthemiddle.
MR-ChiptoChipwithinaPCBA
Figure12MR-ChiptoChip
AninterconnectioninterfacemaybeneededbetweentwochipsonthesamePCBAoronadaughtercardorshortermid-plane.Bydefinition,thisinterfaceisrelativelyshortrangingupto50cm.Thisinterfacecouldincludeasingleconnector.
LR–ChiptoChipacrossaBackplane/Midplane
Figure13LRChiptoChipacrossabackplane
Thisinterfacecommunicatesbetweentwocardsacrossabackplane/midplanewithinachassisandislessthan1mwithupto2connectors.
FECmaybearequirementtomeettheBER–howeverthechoiceoftheFECmustbeconsideredcarefullytoaddressbothlatencyandpowerconcerns.PossibleFECimplementationsincludeRSorBCH.
OFC2016 OpticalInternetworkingForum 24
LR–ChiptoChipacrossaCable
WhilenotanexplicitgoaloftheCEI-56Gproject,cabledversionsofCEI-28Ghavebeenusedtosupportbothbackplaneandchassistochassisinterconnects.
InteroperabilityPointsCEI-56Gdefinesthreedifferentinteroperabilitypointsforthevariousreaches.ThesearethepointsatwhichcompliancethetheOIFIAscanbedeterminedbymeasuringthepropertiesofanimplementationandcomparingthoseresultstotheparametersintheIA.
DieBump
CEI-56G-USRdefinesinteroperabilityatthediebumpofaflip-chipsemiconductordevice
PackageBall
CEI-56G-XSR,CEI-56G-MRandCEI-56G-LRalldefinesinteroperabilityatthepackageball.
Figure14PackageBumpInteroperabilityPoints
OIFchosepackageballsastheinteroperabilitypointsforthesethreespecificationsbecausetheentirechannelistypicallytheresponsibilityofthesamecompany.XSRdoesnothaveaconnectorandthereforenootherinteroperabilitypointisavailable.ForMRandLR,whichdohaveconnectors,thesystemdesignerstillownstheentirechannelandthereforeitdoesnotmakesensetoconstrainhowthesystemdesignerallocatesthelinkbudget.
ModuleInterconnect
CEI-56G-VSRdefinesinteroperabilityatthemoduleinterconnect.
OFC2016 OpticalInternetworkingForum 25
Figure15ChiptoModuleInteroperabilityPoint
Hostcomplianceboards(HCB)andmodulecomplianceboards(MCB)aredefinedtomeasurecompliance.
SummaryTheCEI-56Gfamilyofclausesdefinesasetofinterfacesthatcanbeusedtosolvemosthighspeedinterconnectneeds.Fiveseparatereachesandthreeseparatemodulationtechniquescansupporttheneedsofmostapplications.