512 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 9, NO. 2, MARCH/APRIL 2003
An Optical Centralized Shared-Bus ArchitectureDemonstrator for Microprocessor-to-Memory
InterconnectsXuliang Han, Gicherl Kim, G. Jack Lipovski, Fellow, IEEE, and Ray T. Chen, Senior Member, IEEE
Abstract—An architecture demonstrator of an innovative inter-connect scheme called the optical centralized shared-bus is pre-sented in this paper. This architecture retains the advantages ofshared-bus topology while at the same time specifying a uniforminterface between the electrical and the optical backplane layersin contrast to other proposed architectures. For the first time, afanout equalized optical backplane bus is demonstrated. In this ar-chitecture demonstrator, the data paths required for the micropro-cessor-to-memory interconnects are provided by the optical cen-tralized shared-bus. The optoelectronic interface modules are op-timized to support data rates up to 1.25 Gb/s. The objective of thismicroprocessor-to-memory interconnects demonstration is to en-sure the feasibility of applying this innovative architecture in realsystems.
I NTERCONNECT is becoming an even more dominantfactor in modern computation systems [1]. As the un-
derlying implementation technology, however, electricalinterconnection faces numerous challenges, such as powerconsumption, signal integrity, and electromagnetic interference.The employment of optical interconnects will be one of themajor alternatives for upgrading the interconnect performance[2]. Machine-to-machine interconnection has already beensignificantly improved by utilizing optical means. The majorresearch thrusts in optical interconnects are in the backplaneand board levels where the physical limitations of electricalinterconnects are imposing a prominent bottleneck.
Shared-bus architecture is a preferred interconnect schemebecause its broadcast nature can be effectively utilized to reducelatency, to lessen the complexity of interconnection networkmanagement, and to support cache coherence in a multipro-cessor. However, the physical length, the number of fanouts, andthe speed of the shared-bus are significantly limited by the un-derlying electrical interconnects. Optics has been widely agreedas a better alternative as an interconnect technology. Great re-search efforts have been dedicated to the design of innovative
Manuscript received October 31, 2002; revised February 6, 2003. This workwas supported in part by BMDO, in part by DARPA, in part by ONR, in part byAFOSR, and in part by the ATP program of the State of Texas.
The authors are with the Microelectronic Research Center, Department ofElectrical and Computer Engineering, University of Texas, Austin, TX 78758USA (e-mail: [email protected]).
Digital Object Identifier 10.1109/JSTQE.2003.813311
optical shared-bus architectures [3], [4]. These previously re-ported optical shared-bus architectures, however, are difficult toimplement in practice due to the large variation among the op-tical signal fanouts that dramatically increases the complexityof the optoelectronic interface modules. The difficulty in equal-izing the fanouts is mainly due to the requirement to supportthe bidirectionality of signal flows on the backplane bus. In ourprevious report [5], a new method for rebroadcast signals inan optical backplane bus system was described. Uniform op-tical signal fanouts become possible by using this method. Uponthis method, we present in this paper an innovative architecturecalled the optical centralized shared-bus, which is, to the bestour knowledge, the first fanout equalized optical backplane bus.As shown in Fig. 1, the electrical bus provides interconnectsfor the noncritical paths, whereas the optical bus for the criticalpaths. To implement optical interconnects, volume holographicgratings are used for coupling optical signals into and out of theoptical wave-guiding plate within which optical signals propa-gate as substrate-guided waves [6]. The details of this innova-tive architecture, especially the feature of uniform optical signalfanouts, are described in Section II. In Section III, an opticalcentralized shared-bus architecture demonstrator for micropro-cessor-to-memory interconnects is presented to ensure the fea-sibility of this proposed idea. Finally, a summary is given inSection IV.
II. OPTICAL CENTERALIZED SHARED-BUS ARCHITECTURE
Fig. 1 illustrates the architectural concept of the optical cen-tralized shared-bus. The electrical backplane layer provides in-terconnects for the noncritical paths. The daughter board that isinserted into the central backplane connector plays a pivotal rolein this architecture and is referred to as distributor in this paper.The optoelectronic interface modules, including vertical cavitysurface emitting laser (VCSELs) and photodetectors, are inte-grated on the backside of the electrical backplane and alignedwith the underlying optical backplane layer. Different from theother modules, the positions of the VCSEL and the photode-tector in the central module are swapped. The optical backplanelayer consists of an optical wave-guiding plate with the properlydesigned volume holographic gratings integrated on its top sur-face. Underlying the distributor is a double-grating hologram,and the others are single-grating holograms. In this way, thisarchitecture provides the bidirectionality of signal flows on thebackplane bus.
In this architecture, there are two optical paths for eachsignal line. One is for the source daughter board to deliverthe signal to the distributor, and the other one for the dis-tributor to broadcast the signal to all the daughter boards onthe backplane bus. A complete data transfer from a daughterboard to: 1) another daughter board (point-to-point); 2) morethan one daughter board (multicast); and 3) all the daughterboards on the backplane bus (broadcast), generally involvestwo processes, which are single-hop delivery from the sourcedaughter board to the distributor and then broadcast from thedistributor. This explains the reason we name this architectureoptical centralized shared-bus. First, the VCSEL of the sourcedaughter board projects the light surface normally on itsunderlying holographic grating. This signal is coupled intothe optical wave-guiding plate and propagates within theplate under the total internal reflection (TIR) condition [6].Then, this signal is surface normally coupled out of the plateby the central double-grating hologram and detected by thereceiver of the distributor. Second, the distributor regeneratesthe same optical signal and projects it surface normally on itsunderlying double-grating hologram. This signal is diffractedinto two beams and coupled into the optical wave-guiding plate,propagating along the two opposite directions within the plateunder the TIR condition. During the propagation, a portionof the light is surface normally coupled out of the plate by adaughter board’s underlying holographic grating and detectedby its receiver. This daughter board takes appropriate actionson the received data. If the distributor is the data source, thefirst process will not happen. If the distributor is the only datadestination, the second process is not necessary. The protocolgoverning the bus should be able to tolerate the transmissiondelay due to the difference in the distance. Furthermore, in
conformity with a specific global topology, a hierarchicalinterconnection network [7] can be constructed by using theoptical centralized shared-bus as the building block and thedistributor as the socket.
The most attractive feature of this architecture is to achieveuniform optical signal fanouts. Assuming that the VCSELs ofall the optoelectronic interface modules emit the same opticalpower, uniform fanouts mean that: 1) the power of signals de-livered from any daughter board to the distributor is same; 2)the power of signals broadcast from the distributor to all thedaughter boards on the backplane bus is same; and 3) the powerof signals broadcast from the distributor equals that deliveredfrom any daughter board to the distributor. Thus, this architec-ture specifies a uniform interface between the electrical and theoptical backplane layers in contrast to other proposed architec-tures, e.g., [3] and [4]. On the optical centralized shared-bus, thefanouts are equalized by specifying the diffraction efficiency ofthe volume holographic gratings [5]. Because of the symmetricconfiguration, it is obvious that the two multiplexed gratings in-side the central hologram should have the same diffraction effi-ciency. The analysis in [5] shows that the fanouts are equalizedif the following iterative equation is satisfied:
(1)
where represents the diffraction efficiency of theth single-grating hologram counted from the central double-grating holo-gram. In the case of uniform fanouts, the fanout coefficient,which is defined as the ratio of the fanout power to the effectiveVCSEL fanin power, is obviously equal to the reciprocal of thetotal number of the daughter boards (not including the distrib-utor) on the backplane bus. Thus, the fanout capacity, i.e., the
514 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 9, NO. 2, MARCH/APRIL 2003
maximum number of daughter boards that one optical central-ized shared-bus can accommodate, can be calculated if the biterror rate requirement, marginal power penalty, and the param-eters of the optoelectronic interface modules, such as VCSELemission power and photodetector sensitivity, are specified.
III. M ICROPROCESSOR-TO-MEMORY INTERCONNECTS
DEMONSTATION
As shown in Fig. 2(a) and (b), this architecture demonstratorconsists of a microprocessor board (CPU12, Motorola Inc.),four external memory (8KB SRAM) boards, an electrical back-plane bus, and an optical centralized shared-bus. The micropro-cessor board is inserted into the central backplane connector asthe distributor, and the memory boards, L1, L2, R1, and R2, areinserted into their corresponding slots. Fig. 3 is the conceptualconnectivity block diagram of this system. For simplicity, onlyone memory board is illustrated in this diagram. As shown inFig. 3, the optical centralized shared-bus provides the data pathsfor the microprocessor-to-memory interconnects. Although thesystem clock is only 16 MHz, which is limited by the capacityof the microprocessor (CPU12), this architecture demonstratoris sufficient for our proof-of-concept objective.
A. Volume Holographic Gratings
The volume holographic gratings specified by the op-tical centralized shared-bus architecture were recorded in
Fig. 3. Connectivity block diagram of the architectural demonstrator.
dry photopolymer films (HRF-600X014-20, DuPont). Thephotopolymer-based volume hologram is an attractive can-didate for making high-efficiency gratings. The advantages ofthis material over other types of emulsion, such as dichro-mated gelatin and silver halides, include dry-processing ca-pability, long shelf life, and good photo-speed [8]. Themaximum permissible dose of the incident light is beyond100 JW/cm.
The two-beam interference method was used to form thegratings in the dry photopolymer films. This material consistsof monomers, polymeric binders, and photoinitiators. Themonomers are polymerized when exposed to the light ofspecific wavelength, and the refractive index of the film isdetermined by the polymer concentration. While being exposedto an interference pattern, there are more monomers beingpolymerized in the bright regions than in the dark regions. Thisnonuniform illumination sets up monomer concentration gradi-ents, driving the monomers to diffuse from the dark regions tothe neighborhood bright regions. A final uniform illuminationis required to polymerize the remaining monomers and stabilizethe spatial distribution of the polymer concentration, whichconforms to the original illumination pattern. Thus, a gratingstructure is formed inside the film. To obtain the double-gratingholograms, two sequential exposure steps are required to formthe two multiplexed gratings inside the films.
Our objectives are to obtain single-grating holograms withaccurate diffraction efficiency that satisfies iterative (1), andhigh-efficiency equal-strength double-grating holograms. Thequality of these volume holographic gratings directly affects thefanout variation and capacity, therefore, are pivotal to the im-plementation of the optical centralized shared-bus architecture.The recording schedules were developed by using the methoddescribed in [9]. Following these recording schedules, we wereable to control the accuracy of the diffraction efficiency within2% and obtained 47%/47% double-grating holograms.
Fig. 4 demonstrates the uniform optical signal fanouts inthe architecture demonstrator we built. In conformity with thespecification of the optical centralized shared-bus architecture,a 47%/47% double-grating hologram is integrated in themiddle of the optical wave-guiding plate. The other two aresingle-grating holograms with 50% diffraction efficiency. The
HAN et al.: OPTICAL CENTRALIZED SHARED-BUS ARCHITECTURE DEMONSTRATOR 515
Fig. 4. Demonstration of uniform optical signal fanouts in the architecturedemonstrator.
diffraction angles within the plate are 45, which satisfies theTIR condition. The two 22.5 bevels at both ends are coatedwith aluminum, providing nearly 100% reflection efficiency.An 850-nm VCSEL source was used to obtain the result asshown in Fig. 4. The input optical power from the VCSELwas 2 mW, and the fanouts were, from left to right, 0.404,0.4.06, 0.400, and 0.396 mW, respectively. A two-dimensional(2-D) multibus line configuration [10] can be implemented byreplacing the optical source with a 2-D VCSEL array in thisarchitecture demonstrator.
B. Optoelectronic Interface Modules
The optoelectronic interface modules, consisting of trans-mitters and receivers, implement electrical-to-optical andoptical-to-electrical conversions. As discussed earlier, theoptical centralized shared-bus architecture specifies a uniforminterface between the electrical and the optical backplane layer.A transmitter module consists of an 850–nm VCSEL and alaser driver that accepts differential PECL inputs and pro-vides complementary modulation currents. A receiver moduleconsists of a photodetector/transimpedance amplifier and apostamplifier that accepts a wide range of voltages (10–1200mV) while providing a constant-level (PECL) output voltage.The PCB layout design was optimized to support data ratesup to 1.25 Gb/s. Eye patterns were measured to characterizethe high-speed performance of the optoelectronic interfacemodules in this architecture demonstrator, as shown in Fig. 5.The pseudorandom bit sequence (PRBS) from a pulse generator(HP8183A) was used as the input to a laser driver in an interfacemodule. The output optical signal was transferred through theoptical centralized shared-bus and then detected by a receiverin another interface module. Along with the trigger signalfrom the pulse generator, the output signal from the receiverwas fed into a digital communication analyzer (HP83480A)to display eye patterns. The eye pattern at a data rate of 1.25Gpbs is shown in Fig. 5. This result verifies that these interfacemodules are capable of supporting data rates up to 1.25 Gb/s,thus providing a sufficient headroom for implementing morepowerful computation systems.
Fig. 5. Onboard high-speed performance test and the eye pattern at a data rateof 1.25 Gb/s.
C. Transmitter/Receiver Protocol
To efficiently utilize the large capacity of the optical linksin this architecture demonstrator, serial data transfer was per-formed between the transmitter and the receiver ends. Clockforwarding method, i.e., using the same clock signal at boththe transmitter and the receiver ends, was utilized to implementthis serial data transfer. Without involving complicated clockrecovery circuits, this method reduces the complexity of thesystem. In order to synchronize the serial data with the clocksignal, a special transmitter/receiver protocol was designed. Atthe idle state, the data links keep logic level low [11]. For a se-rial data transfer, the serializer at the transmitter end attachesa logic-high bit in front of the actual data bits, signifying thebeginning timing of the data. Thus, the data pattern presentedon the data links is a logic-high bit followed by the actual datapayload. When received at the receiver end, this logic-high bitsynchronizes the trailing data bits with the clock signal. There-fore, the correct deserializaion can be performed at the receiverend. In this architecture demonstrator, this protocol was carriedby hardware, thus, transparent to the upper programming level.
D. Optical Connectivity Verification
To verify the optical connectivity required by the micro-processor-to-memory interconnects, we ran this architecturedemonstrator under an infinite loop operation. Inside this loop,a memory address was issued to select one memory board,and then a data transfer (write and read back) was performedbetween the microprocessor and the selected memory board.To visualize this data transfer, the data patterns on the opticallinks were displayed on the screen of an oscilloscope, asshown in Fig. 6(a) and (b). In these tests, the data patternswere measured through the monitor ports, as shown in Fig. 2,which indicates the modulation current of their correspondingVCSELs. Fig. 6(a) shows the result when data “04C” wastransferred between the microprocessor and memory boardL1. Along with the starting logic-high bit, the serial datapattern on the optical link should be “100 110 010” in thetime-increasing order. The correct patterns were observed asshown in Fig. 6(a), where channels 1 and 2 show, respectively,the data pattern measured at monitor port L1 and C. Fig. 6(b)shows the result when data “0 44” was transferred between
516 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 9, NO. 2, MARCH/APRIL 2003
Fig. 6. Optical connectivity verification between the microprocessor andmemory boards (a) L1 and (b) R2.
the microprocessor and memory board R2. The correct patternswere also observed as shown in Fig. 6(b), where channels 1 and2 show, respectively, the data pattern measured at monitor portR2 and C. Similar tests were conducted on the optical linksbetween the microprocessor and the other memory boards onthe backplane bus, and the correct patterns were also observed.These results verify the optical connectivity required for themicroprocessor- to-memory interconnects.
IV. CONCLUSION
For the first time, we designed and implemented a fanoutequalized optical backplane bus. This innovative architecture,called the optical centralized shared-bus, retains the advantagesof shared-bus topology, while at the same time, specifying a uni-form interface between the electrical and the optical backplanelayers in contrast to other proposed architectures. The objectiveof this microprocessor-to-memory interconnects demonstrationis to ensure the feasibility of applying this innovative architecturein real systems. The overall performance of this architecturedemonstrator is limited by the microprocessor (CPU12) we used.On the other hand, the optoelectronic interface modules in thisarchitecture demonstrator are able to support data rates up to 1.25Gb/s, thus providing a sufficient headroom for implementingmore powerful computing systems. With the rapid developmentsin active optoelectronic devices, much higher data rate, e.g., 10Gb/s, may be considered in the system design. Furthermore,the optical centralized shared-bus architecture is of generalapplicability and can be applied in scenarios other than themicroprocessor-to-memory interconnects demonstrated herein,e.g., centralized shared-memory multiprocessors.
ACKNOWLEDGMENT
The authors thank BMDO, DARPA, ONR, AFOSR, and theATP program of the State of Texas for supporting this paper.
REFERENCES
[1] W. J. Dally, “Computer architecture is all about interconnect,” inProc.8th Int. Symp. High-Performance Comput. Architecture, Cambridge,MA, Feb. 2002.
[2] M. R. Feldman, S. C. Esener, C. C. Guest, and S. H. Lee, “Comparisonbetween optical and electrical interconnects based on power and speedcharacteristics,”Appl. Opt., vol. 27, pp. 1742–1751, 1988.
[3] J. Yeh, R. K. Kostuk, and K. Tu, “Hybrid free-space optical bussystem for board-to-board interconnections,”Appl. Opt., vol. 35, pp.6354–6364, 1996.
[4] S. Natarajan, C. Zhao, and R. T. Chen, “Bi-directional optical backplanebus for general purpose multi-processor board-to-board optoelectronicinterconnects,”IEEE J. Lightwave Technol., vol. 13, pp. 1031–1040,June 1995.
[5] G. Kim, X. Han, and R. T. Chen, “A method for rebroadcasting signalsin an optical backplane bus system,”IEEE J. Lightwave Technol., vol.19, pp. 959–965, July 2001.
[6] K. Brenner and F. Sauer, “Diffractive-reflective optical interconnects,”Appl. Opt., vol. 27, pp. 4251–4254, 1988.
[7] T. M. Pinkston, “Design considerations for optical interconnects inparallel computers,” inProc. 1st Int. Workshop Massively ParallelProcessing Using Optical Interconnections, Cancun, Mexico, 1994, pp.306–322.
[8] W. J. Gambogi, A. M. Weber, and T. J. Trout, “Advances and applica-tions of DuPont holographic photopolymers,” inProc. SPIE, vol. 2043,1994, pp. 2–13.
[9] X. Han, G. Kim, and R. T. Chen, “Accurate diffraction efficiency controlfor multiplexed volume holographic gratings,”Opt. Eng., vol. 41, pp.2799–2802, 2002.
[10] G. Kim, X. Han, and R. T. Chen, “Crosstalk and interconnection dis-tance considerations for board-to-board optical interconnects using 2-DVCSEL and microlens array,”IEEE Photon. Technol. Lett., vol. 12, pp.743–745, June 2000.
[11] R. T. Chen, “VME optical backplane bus for high performance com-puter,”J. Optoelectron. Devices Technol., vol. 9, pp. 81–94, 1994.
Xuliang Han received the B.S. degree in electronic engineering from TsinghuaUniversity, Beijing, China, in 1999 and the M.S.E degree in electrical and com-puter engineering, in 2001, from the University of Texas, Austin, where he iscurrently pursuing the Ph.D. degree in electrical and computer engineering .
His research interests include massively parallel processing using optical in-terconnects, optical centralized shared-bus, high-speed modular optoelectronictransceivers, guided-wave optics, and holography.
Gicherl Kim received the B.S. and M.S. degrees in physics from Inha Univer-sity, Incheon, Korea, in 1989 and 1991, respectively, and the Ph.D. degree inelectrical and computer engineering from the University of Texas, Austin, in2000.
He was a Researcher in the Agency for Defense Development, Korea from1991 to 1997. He is currently a Research Engineer and Project Manager withOmega Optics, Inc., Austin, TX. His current research topics are mainly de-voted to design and fabrication of optical board and backplane, and the ad-vanced architectural design of optical interconnects and networks. Recently,his research works have also covered the design, fabrication, and packaging ofmodular optical motherboard bus, based on waveguides and substrate-mode in-terconnects using passive and active photonic devices. His interest in the opticaldata transmission topologies includes point-to-point, multidrop, multipoint, andswitch fabric optical communication from the intercomputer to the intracom-puter levels.
HAN et al.: OPTICAL CENTRALIZED SHARED-BUS ARCHITECTURE DEMONSTRATOR 517
G. Jack Lipovski (S’63–M’66–SM’82–F’94) is a Professor in the Departmentof Electrical and Computer Engineering, University of Texas, Austin, where hehas taught since 1976. Previously, he taught electrical engineering and computerscience at the University of Florida from 1969 to 1976. In 1988–1989, he heldthe Grace Hopper Chair of Computer Science, Naval Postgraduate School, Mon-terey, CA. He designed and built the pioneering database computer, CASSM,and parallel computer, TRAC. He has published more than 70 papers and hasauthored or coauthored six books and edited three books. He has consulted forseveral companies, including the Microelectronics and Computer Corporation.He currently studies parallel, database, and artificial intelligence computer ar-chitectures and microcomputers.
Prof. Lipovski chaired the IEEE Computer Society Technical Committee onComputer Architecture and the Computer Architecture Committee of the As-sociation for Computer Machinery. He is also a Computer Society GoverningBoard Member, the Euromicro Director, IEEE MICRO Editor, and the Area Ed-itor of theJournal of Parallel and Distributed Computing.
Ray T. Chen (M’91–SM’98) is the Temple Foundation Endowed Professor inthe Department of Electrical and Computer Engineering, University of Texas,Austin. His research group has been awarded more than 60 research grants andcontracts from such sponsors as the Department of Defense, the National Sci-ence Foundation, the Department of Energy, NASA, the State of Texas, andprivate industry. The optical interconnects research group at UT Austin has re-ported its research in more than 250 published papers. Currently, there are 20Ph.D. degree students and seven postdoctors working in his group. He has servedas a Consultant for various federal agencies and private companies and deliv-ered numerous invited talks to professional societies. His research topics includeguided-wave and free-space optical interconnects, polymer-based integrated op-tics, a polymer waveguide amplifier, graded index polymer waveguide lenses,active optical backplane, traveling-wave electrooptic polymer waveguide modu-lator, optical control of phased-array antenna, GaAs all-optical crossbar switch,holographic lithography, and holographic optical elements.
Prof. Chen has chaired or been a Program Committee Member for more than40 domestic and international conferences organized by SPIE, OSA, IEE, andPSC. He is a Fellow of SPIE and the Optical Society of America and a Memberof PSC.
