High Performance Optical Interconnect Project

Final Report

May 07-06

Client: Lockheed Martin

Advisors: Dr. Arun Somani, Dr. Mani Mina, Dr. Robert Weber

Team Members: David Sheets, Jay Becker, Adam Fritz,

and Layth Al-Jalil

DISCLAIMER NOTICE! DISCLAIMER: This document was developed as a part of the requirements of an electrical and computer engineering course at Iowa State University, Ames, Iowa. This document does not constitute a professional engineering design or a professional land surveying document. Although the information is intended to be accurate, the associated students, faculty, and Iowa State University make no claims, promises, or guarantees about the accuracy, completeness, quality, or adequacy of the information. The user of this document shall ensure that any such use does not violate any laws with regard to professional licensing and certification requirements. This use includes any work resulting from this student-prepared document that is required to be under the responsible charge of a licensed engineer or surveyor. This document is copyrighted by the students who produced this document and the associated faculty advisors. No part may be reproduced without the written permission of the senior design course coordinator.

May 2, 2007

- 2 -

Table of Contents ITEM PAGE 1 Introduction............................................................................................................... 9

1.1 Executive Summary .......................................................................................... 9 1.1.1 Project Task............................................................................................... 9 1.1.2 Project Activities ....................................................................................... 9 1.1.3 Final Results ............................................................................................ 10 1.1.4 Recommendations ................................................................................... 11

1.2 Acknowledgement ........................................................................................... 11 1.3 Problem Statement.......................................................................................... 11

1.3.1 The Problem ............................................................................................ 12 1.3.2 The Solution............................................................................................. 12

1.4 Operating Environment .................................................................................... 13 1.4.1 Laboratory Operating Environment..................................................... 14 1.4.2 Deployment Operating Environment.................................................... 14

1.5 Intended User(s) and Use(s) ........................................................................... 14 1.5.1 Intended User(s) ...................................................................................... 14 1.5.2 Intended Use(s)........................................................................................ 14

1.6 Assumptions and Limitations ........................................................................ 15 1.6.1 Assumptions............................................................................................. 15 1.6.2 Limitations............................................................................................... 15

1.7 Expected End Product ...................................................................................... 15 1.7.1 Intellectual Properties ............................................................................... 15 1.7.2 System Hardware........................................................................................ 16 1.7.3 System Software ......................................................................................... 16

2. Product Approach and Results................................................................................ 17 2.1 Approach Used ................................................................................................ 17

2.1.1 Functional Requirements ....................................................................... 17 2.1.1.1 Topology............................................................................................... 17

2.1.1.1.1 Transmission ................................................................................. 18 2.1.1.2 Board / Fiber Optic Interface ............................................................ 19 2.1.1.3 COTS.................................................................................................... 19 2.1.1.4 Data Encoding ..................................................................................... 19 2.1.1.5 Network Control ..................................................................................... 19

2.1.2 Design constraints ................................................................................... 20 2.1.2.1 Topology............................................................................................... 20 2.1.2.2 Multiplexing......................................................................................... 20 2.1.2.3 Transmission ........................................................................................... 21 2.1.2.4 Board / Fiber Optical Interface ............................................................. 21

2.1.3 Technical Approach Considerations ..................................................... 21 2.1.4 Testing Approach Considerations ......................................................... 22 2.1.5 Design Requirements .............................................................................. 22

2.1.5.1 Decentralized Control............................................................................. 22 2.1.5.2 Fault Tolerance ....................................................................................... 23 2.1.5.3 Expandability .......................................................................................... 24

2.2 Detailed Design...................................................................................................... 25

- 3 -

2.2.1 Hardware Design ........................................................................................... 25 2.2.1.1 Point to Point Interconnection ............................................................... 25 2.2.1.2 Unidirectional Bus with Single Fiber .................................................... 26

2.2.1.2.1 Parts List........................................................................................... 26 2.2.1.3 Bidirectional Ring with Single Fiber..................................................... 26

2.2.1.3.1 Coupler/Splitter Sub-network of Bidirectional Network ............. 27 2.2.1.3.1.1 Scattering Parameters .............................................................. 27

2.2.2 Software Design.............................................................................................. 29 2.2.2.1 Basic TDM Scheme ............................................................................. 29

2.2.2.1.1 Primary Controller .......................................................................... 30 2.2.2.1.2 Secondary Controller....................................................................... 30 2.2.2.1.3 Controlled ......................................................................................... 30

2.2.2.2 Advanced TDM Scheme ......................................................................... 30 2.2.2.3 Aurora Protocol ...................................................................................... 31

2.3 Implementation Process description ............................................................. 32 2.4 Testing of the End Product and its Results .................................................. 32

2.4.1 Lab Equipment Utilized ......................................................................... 32 2.4.2 Software Used.......................................................................................... 32 2.4.3 Fiber Optic Testing ................................................................................. 33 2.4.4 Software Testing...................................................................................... 33

2.4.4.1 Basic Communication Between Two Processor Cores .................... 33 2.4.4.2 Basic Communication Between Two Boards.................................... 34

2.4.4.3 Connection Between Boards ........................................................... 34 2.4.5 System Integration .................................................................................. 34

2.5 End results of the project ............................................................................... 34 3. Resources and Schedules......................................................................................... 35

3.1 Resources ......................................................................................................... 35 3.1.1 Personnel.................................................................................................. 35 3.1.2 Other Required Resources ..................................................................... 37 3.1.3 Project Costs............................................................................................ 38

3.2 Schedules.......................................................................................................... 40 3.2.1 Project Schedule...................................................................................... 40 3.2.2 Deliverables ............................................................................................. 42

4. Closure Materials..................................................................................................... 42 4.1 Project evaluation ........................................................................................... 43 4.2 Commercialization .......................................................................................... 43 4.3 Recommendations ........................................................................................... 43

4.3.1 Project Continuation .................................................................................. 43 4.3.1.1 Signal Filtering .................................................................................... 43 4.3.1.2 The NIC................................................................................................ 44 4.3.1.3 Software Implementation ................................................................... 44 4.3.1.4 Signal Throughput .............................................................................. 45

4.4 Lessons Learned.............................................................................................. 45 4.5 Risk and Risk Management ........................................................................... 45 4.6 Project team information ............................................................................... 45 4.7 Closing Summary............................................................................................ 47

- 4 -

4.8 References ........................................................................................................ 47 4.9 Appendix.......................................................................................................... 48

4.9.1 Appendix A .............................................................................................. 48 4.9.2 Appendix B .............................................................................................. 52 4.9.3 Appendix C .............................................................................................. 53

- 5 -

List of Tables

Table #1: Parts List..........................................................................................................26 Table #2: Original Personnel Effort Requirements (hours) ........................................36 Table #3: Revised Personnel Effort Requirements (hours) .........................................36 Table #4: Final Personnel Effort Requirements (up to April 1, 2007)........................37 Table #5: Original Other Resource Requirements .......................................................38 Table #6: Revised Other Resource Requirements .......................................................38 Table #7: Final Other Resource Requirements.............................................................38 Table #8: Original Financial Costs.................................................................................39 Table #9: Revised Financial Costs..................................................................................39 Table #10: Total Final Financial Costs ..........................................................................40 Table #11: Project Team Information ...........................................................................41

- 6 -

List of Figures

Figure 1: Bus Topology ...................................................................................................13 Figure 2: Unidirectional Setup .......................................................................................16 Figure 3: General Bus Topology.....................................................................................17 Figure 4: Transmission Illustration ...............................................................................18 Figure 5: Decentralized Control .....................................................................................23 Figure 6: Quasi-decentralized Control ..........................................................................23 Figure 7: Fault Tolerance................................................................................................24 Figure 8: Expandability...................................................................................................25 Figure 9: Point to Point Interconnect.............................................................................25 Figure 10: Unidirectional Bus Hardware Design..........................................................26 Figure 11: Bidirectional with single fiber ......................................................................27 Figure 12: Bidirectional Sub-network ...........................................................................27 Figure 13: S-Parameters/Splitter....................................................................................28 Figure 14: S-Parameters Coupler...................................................................................28 Figure 15: S-Parameters Overall....................................................................................29 Figure 16: Basic TDM Scheme .......................................................................................29 Figure 17: Advanced TDM Scheme layout....................................................................30 Figure 18: Original Project Schedule.............................................................................40 Figure 19: Revised Project Schedule..............................................................................41 Figure 20: Final Project Schedule ..................................................................................41 Figure 21: Deliverables Schedule ...................................................................................42 Figure 22: High level TDM illustration ........................................................................48

- 7 -

List of Definitions Aurora – A simple protocol used in signal transmission Application layer - provide helping services to running processes and usually where the processes located, for example, the browser process uses HTTP application layer protocol to access the World Wide Web (WWW) ADS – Electronic design automation software widely used in RF area Backplane – A stationary transmitting material, whose available connections may or may not all be used Board - A circuit card with processing, memory, and input/output capability CDMA – Code division multiple access COTS – Commercial off-the-shelf EMI – Electromagnetic interference End to end protocol - it is a protocol responsible for successful logical end – to – end transfer of date between two processes running in the source and destination machines where the transmission is independent of lower layers usually through sockets use. (Transmission Control Protocol – TCP is an example) Ethernet – A family of networking technologies for local area networks FPGA – Field programmable gate array – A reprogrammable chip Frame - is a block of information (a packet) that is arranged in a particular manner for transmission between two machines LEG- is the combination of nodes and fibers that provide a conduit for fiber optic signals without such a conduit being a full loop; akin to an open circuit LAN – Local area network LRU – Line replaceable unit (interconnected with cabling) LRM – Line replaceable module (interconnected with a common backplane) MAC - Medium access control, a procedure that coordinates the transmission of data over the medium of communication Modal distortion – A type of distortion where a signal spreads with respect to time Moore's Law - the empirical observation that the transistor density of integrated circuits, with respect to minimum component cost, doubles every 24 months, attributed to Gordon E. Moore, a co-founder of Intel Multi mode fiber – Fiber optic cable that transmits lights through any of a number of different paths down the length of the fiber Multiplexing – combining multiple sources of information into one Network - The interconnection of boards, constituent equipment, and functional software. Synonymous with "system", however has the connotation of how the network operates, not how the system is used

- 8 -

NIC – Network interface card Node – A point of connection to the Optical Interconnect OSI - (open system interconnection) reference model, it presents the basic seven functions required for two computers to communicate, they are: Application, Presentation, Session, Transport, Network, Data Link, and Physical layers Parallel Communication– Signals traveling in multiple wires PCI-X: It is a computer bus standard that developed to expand the speed of early developed PCI (Peripheral Component Interconnect) PCI-E (PCI Express): It is a computer bus standard that is based on serial non shared communication lines Presentation layer - provide application layer independency service from differences of data representations of layers below. (Not used in TCP/IP networks) Protocol – a set of rules governing communication between electronic devices RapidIO – a high-performance, packet-switched, interconnect technology RF – Radio frequency RISC – Reduced instruction set computer Routing protocol - is a procedure that used to select paths across various networks (Internet Protocol – IP is an example) SATA - A computer bus technology primarily designed to transfer data to and from disk drives Serial – Signals traveling in a single wire Session Layer - control the way in which data are exchanged, e.g. half or full duplex. (Not used in TCP/IP networks) Single mode fiber – Fiber optic cable that transmits light through a single path SMA - Coaxial RF connectors to interface with coaxial cable System - The interconnection of boards, constituent equipment, and functional software; Synonymous with "network", however has the connotation of how the system is used, not how the network operates TDM – Time division multiplexing – Each component of the system gets a fraction of the available time to transmit, and listens the rest of the time Transducer – A device that converts one type of energy into another Wavelength – the distance between repeating units of a wave pattern WDM – Wave division multiplexing – Signals travel at different wavelengths allowing multiple transmissions as one time LAN PHY – A standard that defines Ethernet transmission between routers and switches

- 9 -

1 Introduction The project will be introduced through an executive summary, coupled with acknowledgements, problem statement and proposed solution, proposed operating environment, intended user(s) and use(s), limitations and assumptions, end product description, and other deliverables. 1.1 Executive Summary The following section will bring the reader up to speed on the general details of this project by providing an overview of the project, project activities, final results, and recommendations for future work. 1.1.1 Project Task The May07-06 Senior Design team has been tasked with developing an approach to interconnect 2 to 4 processors using 10Gbps technology and consumer-off-the-shelf equipment where data is transmitted via fiber optics. However, due to the $5000 budget set for this project, the team was instructed to develop the most capable system within the budget constraints. The cost of the initial system forced reconsideration of the problem definition. In order to avoid breaking the $5000 barrier, the team reconstructed the project task to develop a similar system operating at approximately 2.4Gbps. This system would utilize Xilinx Virtex II boards that are located in the high speed systems engineering laboratory at Iowa State University as well as 1310nm fiber optic transceivers that are available through the electrical engineering department. A development platform was constructed using the available lab equipment in order to gain an understanding of how to interconnect processors via fiber optics. This project will serve mainly as a research project which will aid future design teams at Iowa State University and Lockheed Martin in designing fiber optic networks. 1.1.2 Project Activities During the first semester of this project, the team completed several milestones including the following:

• Problem definition: Develop a fault tolerant, distributed control, fiber optic network that supports data transfer at 10 Gbps. Fault tolerance, expandable, and distributed control were designed to be achieved through a fiber optic ring supporting bi-directional data transfer.

• Technology considerations and selections: Due to the $5000 budget limitation that has been set for this project, the purchase of 10 Gbps fiber optic NICs was not feasible. As a result, the team chose to implement a similar network running at

- 10 -

approximately 2.4Gbps which takes into consideration the common issues associated with 10Gbps communication.

• Prototype design: The prototype was designed to be a fiber optic bus wherein each node was a combination of transceiver, NIC, and passive fiber optic circuitry.

• Project Documentation

This semester the team has made considerable progress on the following milestones:

• Prototype Implementation: A prototype using Xilinx Virtex Pro II boards (2.4Gbps), multimode transceivers and passive circuitry has been demonstrated in the lab to the advisors. The development of the prototype was broken into two components.

o The first component involved establishing reliable communication between multiple processing boards via SATA cabling as well as developing an organized communication scheme that would manage the signal transmissions across the network.

o The second component involved designing and testing a coupler/splitter configuration that serves the purpose of interfacing each processing board with the fiber optic medium that connects all the boards in the network. The final design includes two 50/50 couplers at each node to route signals coming from either direction on the network to and from the node.

o After researching the limitations of a fiber optic network, it was decided that a ring topology was not feasible without signal termination on the network. Therefore, a bus topology was implemented where signals would be terminated at the end of the bus.

• Documentation and Presentation (Class presentation, project poster, final report, IRP presentation, as well as a summary of the performance results obtained)

1.1.3 Final Results The end product is a network of processors operating at approximately 2.4Gbps which utilizes a bus topology and a fiber optic communication medium. This is accompanied by this report summarizing performance results of such a system. A computer is used to interface with the NIC(s) in order to develop and deploy code as well as testing and displaying transmission results. Currently, the architecture consists of a computer and a network interface card linked by a USB cable. Data is sent over the USB cable at roughly 480 Mbps. The data is then sent out on the network to the desired node or nodes. All nodes are connected similarly to a computer as a user interface. Transmission on the network is governed by one board which is instructed by the “researcher” when to transmit data, what data is to be transmitted, and where to transmit the data. The architecture selected for this project first employed a point-to-point topology using two processing nodes which was developed into a bus topology in order to resolve

- 11 -

technical issues. Both point-to-point and bus topologies involve interfacing fiber optic NIC(s) with PC(s). Point-to-point topology involves the direct interconnection of two fiber optic NIC(s) via fiber optic medium without additional passive components (i.e. fiber optic couplers, splitters, etc.). Building upon the former architecture, the bus topology involves a fiber optic “disconnected loop” wherein every node has a coupler/splitter configuration that drives signals onto and couples signals off of such a loop. These interconnected processor boards are to be utilized as a research tool for fiber optic network designers to learn about the capabilities and limitations of high speed communication. The purpose is to provide a lightweight, EMI insensitive, high speed interconnect in place of heavier, slower, and EMI sensitive cabling. The transmission rate for this solution is 2.4Gbps. 1.1.4 Recommendations Future teams could make several improvements to the project including:

• Increase network speeds with proper trancievers • Replace Aurora with a more robust protocol • Migrate to a PCI-E based NIC • Implement a more advanced scheduling scheme, possibly prioritizing nodes

1.2 Acknowledgement The group recognizes the contribution of Drs. Somani, Mina, and Weber to the technical understanding of fiber optic communication and processor networking. Specifically, Dr. Weber has provided great insight into the problems of fiber optic communication; Dr. Mina has provided great insight into the problem of moving information from an electronic signal to an optical signal; and Dr. Somani has provided guidance toward the approach that should be taken when building a network of processing nodes. The group further recognizes Nathan VanderHorn, Michael Frederick, and Rashmi Bahuguna, who are Ph.D. candidates working in the Dependable Computing and Networking Laboratory. Also, Jason Boyd, Lab Coordinator in computer engineering, has assisted the group in formulating the current design approach. Kevin Finck, senior in electrical engineering at ISU, also has contributed to this project with suggestions. Rick Stevens has provided a general understanding on the needs of Lockheed Martin on this project and Aaron Cortes, ISU Alumni, former student for Dr. Somani, and Lockheed Martin employee, have both provided the necessary information for this team to continue developing the architecture of this project. 1.3 Problem Statement Today’s high speed communications industry constantly seeks improved throughput as is required for increasingly complex systems. Increases in throughput demand on all

- 12 -

communication systems stem from the sheer bulk of information being transferred between elements of any given network. This project is aimed towards avionics applications where the network is the interconnection of all the computers on board.

1.3.1 The Problem When computer processing is distributed throughout avionic platforms, the ability of processors to communicate at accelerated speeds greatly enhances the functionality of one’s platform. The task of this project is to develop a network of processors that can be used to research the limitations and possibilities of high speed communication. The need arises to separate computer processing, and consequently utilize high speed communication concepts when:

• Electrical domain hardware cannot provide the reliable throughput of fiber optic

communication. Reliable, in this case, means the ability to withstand EMI effects and other signal degradation. However, most current avionics systems do not natively support fiber optic traffic. Thus an interface must be devised.

• Moore’s Law fails to keep up with increases in processor ability. Modern avionics software drives a demand for increasing processor speed, thus a mitigation strategy involves increasing processor power by binding multiple processors to perform a given operation.

Today’s avionics marketplace wants to purchase solutions that allow for greater throughput between hardware elements in a system. Currently, gigabit Ethernet is the most commercially available "fast" solution for the problem of system interconnectivity. However, the need to push more data faster between points in an avionics network exasperates the engineering limits of transmitting data electronically. 1.3.2 The Solution Fiber optic standards provide a consumer-off-the-shelf (COTS) solution to the issue of transferring data point to point within a platform. While fiber optic hardware and data encoding are in principle dissimilar, for engineering ease, standards already exist for the transmission of data on a fiber optic line. Lockheed Martin has instructed the May07-06 Senior Design Group to compose a system of at least two NIC(s) interconnected with fiber optic lines and a switch or switches to facilitate a time division multiplexing scheme between NIC(s). A similar system has been developed by the team to potentially support up to 4 NIC(s). As reflected in the executive summary, processor to processor transmission of 10Gbps was not possible under the budget constraints in place. However, board-to-board communication with transmission rates of approximately 2.4Gbps has been achieved using SMA cable to connect each node. Upon completion of satisfactory testing of the

- 13 -

time division multiplexing scheme that has been designed, each node is linked with multimode fiber optic cable. The TDM multiplexing scheme that is used to divide time between the processing nodes has been designed and is in the initial stages of implementation. See Section 2.1.1.2.

Figure 1: Bus Topology

The solution approach that is discussed in this document is shown above. It illustrates the bus topology of the network of processors. The data transfer protocol selected for this network is Aurora. Aurora is a simple serial data transfer protocol that uses 8-bit-to-10-bit encoding or 64-bit-to-66-bit encoding. The particular version chosen uses 8-bit/10-bit encoding and is done so that the clock signal is sent along with a packet so that the transmitter and receiver can synchronize clocks. This format is also chosen because of ease of use. Each node in the network also contains a fiber optic coupler/splitter configuration, which directs some signal power into the NIC receiver as it circulates through the bus and passes the remainder of signal power on to the next node in the network. Such a configuration also evenly splits power in both directions of the bus as a particular NIC transmits a signal. This configuration has been designed, tested, and verified to be able to perform these functions. 1.4 Operating Environment The operating environment drives the selection of parts, software, and topology. The operating environment can be the platform on which the system operates, the ambient conditions affecting the system, or the effect users may have on the system.

������

�����

� �

�

�

�

�

������

�����

� �

�

�

�

�

������

�����

� �

�

�

�

�

- 14 -

1.4.1 Laboratory Operating Environment Development and testing of the system requires a lab environment where boards can be integrated and removed from the network easily and controlled tests can be performed to confirm system fidelity. The boards are linked together and each board sits as an interface between two different nodes. 1.4.2 Deployment Operating Environment Lockheed Martin’s end use for this product is to set a board in between LRU’s and LRM’s within a given platform or vehicle and use the board to transfer slower electrical data transmitted out of a given box into much faster fiber optic signals. The operating environment of such a solution will be military aircraft and other military vehicles. Such a system will likely be derived from the reports made during the project detailing the performance of the system. 1.5 Intended User(s) and Use(s) The intended user(s) and intended use(s) are interdependent but for the purpose of analysis each should be considered independently. This is particularly true for passive users who may not be aware they are using this system. 1.5.1 Intended User(s)

The intended user of this system is a pilot who is operating an airplane with avionic equipment interconnected using this system. This user will not directly interface with the hardware but will be directly affected by this equipment, both in the sense that system latency will be decreased as a consequence of this system, and also that failure of the optical interconnect will at best mean reverting to older equipment and at worst mean complete system failure. The user with more direct interaction with the optic interconnect will be engineers at Lockheed Martin and researchers/students at Iowa State University. Their interaction with the optical interconnect will be to directly manipulate the network for research purposes and not necessarily rely on the system to maintain system operability. 1.5.2 Intended Use(s)

The first and principle use of the fiber optic network is to link LRU’s and LRM’s together within an avionics platform and to facilitate accelerated data transfer therein. Whereas prior to the deployment of this interconnect an LRU may have to transmit at a slower speed due to bus or transmission limitations. The proposed interconnects speed greatly exceeds the processing speed and the transmission speed of most LRU’s. This raises the limit on how fast an LRU can transmit.

- 15 -

Secondary to its use as a backbone between LRU’s, this system will also be used by researchers at Iowa State and by engineers at Lockheed Martin as a means of testing the viability of different hardware protocols, processor/memory configurations, and fiber optic technologies as a means of solving interconnect problems in future avionic systems.

1.6 Assumptions and Limitations This section encompasses project level constraints at the beginning of the project. 1.6.1 Assumptions • Lockheed Martin will provide a budget of $5000 for purchase of COTS equipment. • There will not be a requirement to develop a board from scratch; the team will use

Xilinx Virtex II boards, or equivalent COTS equipments. • Processor boards will be provided by either Iowa State University or by Lockheed

Martin. • Interconnect equipment will be purchased by May07-06 Senior Design Team. • Interconnect system will be plug and play compatible. • Development of this system will be largely drawn from existing research being done

by the Dependable Computing and Networking Laboratory. • VHDL software exists for the available Xilinx Virtex II boards in the lab. 1.6.2 Limitations

• The duration of the project is 8 months. • The budget for the project is $5000. • While the final system will involve boards being directly integrated into LRU’s, this

project will have stand alone boards that are not integrated into a platform. • Students cannot commit to this project full time, thus causing some tasks to be

prolonged and done poorly. Poor work requires group oversight, and thus more time. • There are approximately twenty multi-gigabit transceivers on the market today. • The distance between nodes using multi-mode fiber optic cabling cannot exceed 300

meters. • Max number of nodes is 4. • In order to add nodes to the network, a more powerful Tx module is required. 1.7 Expected End Product The end product of this project can be broken into intellectual properties, fiber optics, and network interface. 1.7.1 Intellectual Properties With the hardware and software system currently being tested, this report has been created to describe the results of the experimentation. This report, along with all project

- 16 -

documentation, will be furnished to the customer, Lockheed Martin, as an end product of this senior design project. 1.7.2 System Hardware

• Computing module: This interfaces with the fiber optic NIC via USB and RS232. • Fiber optic NIC: This performs the electro/optical conversion and encoding into a

shared protocol. • Fiber optic couplers and splitters: These components direct the propagation of

power across the bus. • Fiber optic cable: The transmission medium.

This product is comprised of an NIC that houses a processor, memory, and fiber optic transceivers. A USB bus inputs from the host data that is transferred to an NIC. Next in the sequence, a laser transceiver converts the serial signal brought out of the NIC electrical circuitry into an optical signal. Fiber optic cabling and passive components interconnect one board to another board.

Figure 2: Unidirectional Setup 1.7.3 System Software There are two different areas that make up the system software: the VHDL that describes the hardware, and the program that runs to control the hardware to ensure reliable node-to-node transmission. The bit stream generated from the VHDL is downloaded to each Xilinx development board that represents a node. No time division multiplexing scheme has been implemented as of yet. However, all TDM design details have been included in appendix A.

������

�

�

- 17 -

2. Product Approach and Results This section will discuss the details of the design and results of the implementation as the project has progressed.

2.1 Approach Used The proposed approach is derived from Section 1.3.2. This approach uses a bus topology to ensure reliable data transfer between nodes while avoiding fiber saturation. Fiber saturation is the result of inadequate signal termination across the network.

2.1.1 Functional Requirements Functional requirements are driven by the overall requirement of a portable fiber optic network that transfers data between nodes at three gigabits per second. 2.1.1.1 Topology

The bus topology, specified in Section 1.3.2, involves arranging nodes in a disconnected loop configuration. The following diagram shows the concept of the bus topology and the directivity of data transfer.

Figure 3: General Bus Topology

Each node shall consist of two fiber optic couplers, encapsulated above as “node n.” Between each node (and between splitters and couplers comprising a node) is fiber optic cable.

������

�

�

- 18 -

Several factors affect the each topology:

• Power level of signals incident to a particular node based on the path taken through the topology.

• Distance between nodes in the final design. • How often a node is ready to transmit. • Whether a node, because of inter-node signal attenuation, has access to the master

node. These issues are explained hence. 2.1.1.1.1 Transmission

Node to node data transmission shall be done at 2.4 gigabits per second. The medium of transmission shall be multimode mode fiber optic line.

Figure 4: Transmission Illustration After experimentation with a loop topology, a problem was found with the signal energy on the fiber optic cabling. As signals were sent out on the network and propagated from node to node, there was no way other than attenuation to terminate the signal. Therefore, until the signal power had dissipated to a level that wasn’t recognizable, the signal would continue propagating around the loop. After multiple signals had been transmitted across the network, the fiber optic cable became saturated with signal energy. The transmitted data was then unreadable. To solve this issue, a bus topology was chosen so that signals transmitted in both directions from a given node would be terminated at the ends of the bus. Reliable data transfer was then restored.

������

�

�

- 19 -

Once a signal is transmitted, the signal is attenuated both by the fiber in which it travels and also by every node in the network as a consequence of propagating through the coupler/splitter configuration. Transmitted signals continue to propagate until terminated or entirely dissipated in the fiber.

2.1.1.2 Board / Fiber Optic Interface The NIC is a Xilinx Virtex II Pro board wherein the fiber optic transceiver is off board. The fiber optic interface is made through the SMA connectors on the board. Differential RF signals are brought to the transceiver module, which therein drives the fiber optic transceiver. Care should be taken that for either the receiver pair or the transmit pair a differential pair of signals be transmitted over two lines of the same length. 2.1.1.3 COTS

All equipment used to build this system is consumer-off-the-shelf equipment.

2.1.1.4 Data Encoding NIC(s) interface with the host computer over a high speed USB port, 480 Mbps. Data is transmitted in the Aurora format. Since the Aurora specification provides for a synchronization signal, such must be deactivated before nodes are connected to the fiber optic network. Without this synchronization signal, the header for a packet must include a component for synchronization. 2.1.1.5 Network Control Due to the magnitude of this project and budget constraints, network control issues were not sufficiently experimented with. In follow-on projects this section would be useful for designing a network with decentralized control. Philosophically, meeting the requirement of decentralized control invokes an image of every node determining how to control itself during a given time period and as such every node in a network must reach the same conclusion independently. Without great care, this is not a fault tolerant solution. A fault tolerant solution is to assign a master node (stored in memory and loaded on startup). On power up, the master node takes a survey of the network and sees if it can transmit in both directions; because attenuation is still an issue, the master will transmit, for this interval, at maximum power, just to determine if it can read its signal. Once that is determined, then the master will rebroadcast its ping signal, bringing signal intensity down until the master can no longer read its signal. Then the master begins pinging every node in the network, ascertaining delay time and the number of nodes.

- 20 -

If the ring is not functioning and the master cannot perceive its own signal on the first try, the master will assume that the ring functionality is down and shall continue, pinging every node on the modified "ring", which is now, effectively, a directional bus.

2.1.2 Design constraints

Constraints are generally a result of budget and time limitations, and technological limitations.

2.1.2.1 Topology

Topology hinges on one node being able to transmit information to all other nodes without significant propagation delay. As nodes are added to the topology, the power of the fiber optic signal will degrade. Thus, there is an upper bound on nodes which is dependant on the receiver chosen to transfer optical information into RF information. With reference to a transmitting node, a node degrades the signal that appears at the next farthest node. This is because each node requires some optical energy to be routed into that node's transceiver to ascertain whether information being transmitted is valid for that node. The topology was also limited by the lab equipment available to the design team. Currently, it is feasible to design four nodes at 1310nm and/or three nodes at 850nm using available lab equipment.

2.1.2.2 Multiplexing Due to the magnitude of this project and budget limitations, extensive experimentation with TDM multiplexing was not accomplished. Future design teams should consider the TDM section of appendix A as well as the following section in designing multiplexing schemes. The initial implementation of TDM involves giving each node an equal amount of time to transmit. This means that as nodes are added, each node is given less time to transmit. There is therefore a bound on each node as to the size of any information packet it wishes to transmit. Subsequent implementations of TDM will involve greater fairness, in that nodes will be allotted time based on the amount of data to be transfer by that node. The current system uses one wavelength for data transfer.

Code division multiple access is a viable option except the complexity of the solution is outside the scope of this project. Waveform division multiplexing may be a preferable option if one wishes to let multiple boards talk simultaneously. This as well is outside the scope of this project.

- 21 -

2.1.2.3 Transmission Transmission of a 3Gbps signal requires hardware to sit between the board and the laser transducer to serialize a parallel signal coming off the board unless the board generates a 3 GHz signal natively. Any laser transducer driven by RF should be able to switch on faster than 0.05 ns and switch off faster than 0.05 ns. This nets a maximum period for a bit as 0.1 ns, which is the period of a signal whose frequency is 10 GHz. Furthermore, as signals propagate down the fiber optic medium, signal power is lost to dispersion, necessitating additional signal processing to correct for lost signal intensity. 2.1.2.4 Board / Fiber Optical Interface Transmission distance through multi-mode fiber optic medium cannot exceed 300 meters for desired 3Gbps transfer rate. However, this is not a problem because it is assumed that the distanced between nodes in an avionics platform will not exceed 300 meters. Transmission over significant distances using multi-mode fibers must utilize optical wavelengths between 650 and 750 nm. Use of multi-mode fiber introduces the prospect of added signal distortion due to modal distortion. 2.1.3 Technical Approach Considerations The issue with fiber optic communication is that optical signals must be converted into electrical signals before information is usable. This drives a latency issue, and as such the design does not make any particular node dependent on optical-electrical conversion on any other node, save the fact that the transmitter is driven by an electrical signal. When taking the loop topology into consideration, because there is no signal termination in unidirectional loop architecture or in bidirectional single loop architecture, the output power of the transmitter must be adjusted so that once its own signal returns to itself the signal is attenuated to appear as noise. The limit on the number of nodes in the final design is four. Reliable data transmission becomes increasing complex as more nodes are added. Because of the limited budget, the design must also integrate nodes into the network that are not necessarily fiber optic; to support such, a bridge must exist between them. This bridge may well be using another PCI-e port on a host computer and moving information from a fiber optic card to a non-fiber optic card.

- 22 -

2.1.4 Testing Approach Considerations Prior to prototype demonstration, all components (hardware and software) will be tested thoroughly. The testing procedure was as follows: Step 1: A single board has software loaded onto it which specifies how the FPGA (specifically the on board processor) buffers incoming information and how such information is transmitted off the board. This step will involve transmitting a file into the system externally and transmitting such data out. Since this is a loopback test, the information comes back into the card and is processed like received data. Thus, the test is to ensure data transmitted is the same as data received. Step 2: Two boards were then connected using SATA cabling/SMA cabling to test the

software algorithms that ensure reliable communication. Step 3: Step 1 is repeated except that between the transmitter and receiver port a laser transducer is be placed to emit an optical signal out and to transfer optical information into an RF signal. Step 4: Step 2 is repeated with the caveat that another board is introduced and monitored to measure the fidelity of information transferred between boards. 2.1.5 Design Requirements There are three governing philosophies that govern the design of the fiber optic network: decentralized control, fault tolerance, and expandability. For future improvement on the research of this project the following sections should be taken into consideration. The following three sub-sections (decentralized control, fault tolerance, and expandability) have not been implemented into the development platform chosen for this project. 2.1.5.1 Decentralized Control Decentralized control refers to scheduling each node time to talk on the network. The scheduling process itself is time division multiplexing (TDM). This is a process whereby each node on the network is assigned a partition of time to talk out of a predetermined period wherein each node is assigned time based upon relative need. True decentralized control would require every node on the system to perform an analysis based on data transmitted by every other node on the system as to both how much time a specific node is allotted as well as determining the order of transmission of every card (i.e. whose information to expect, when to expect it, and when to expect to transmit) and being synchronized with the rest of the network.

- 23 -

Figure 5: Decentralized Control

Quasi-decentralized control will involve having a master which determines when every node gets to talk and for what duration. Such a master could be replaced with a predetermined back-up master; this assignment could be both defaulted and also assigned by the master itself. A natural extension of this concept would be for every node in the network to be a potential master, predetermined by the previous master. Assignment of the heir node would be done by the current master only when that node became the master.

Figure 6: Quasi-decentralized Control

2.1.5.2 Fault Tolerance Fault tolerance is represented in quasi-decentralized control, where a centralized network can be rebuilt as another centralized network if the master node is taken down. Another aspect of fault tolerance is the bi-directional nature of a fiber optic network. Utilizing ring architecture will allow each node to transmit a fiber optic signal in both directions around the loop. This will allow the network to tolerate one failure in the network and continue to operate. Spanning-tree architecture was not used because of the issue of signal termination and determination of what spanning tree to use to interconnect two nodes. Given that every

- 24 -

node on the network is a potential recipient of a fiber optic signal, a signal between two points that are not adjacent on the network requires a unique spanning tree (albeit a great deal of any given path may be reused by another inter-nodal transmission), and such is contingent on terminating the fiber optic signal, least such continues to erroneously propagate throughout the network. One such method of termination is to convert a signal from optical to electrical at the node and then determine if the signal was intended for that specific node; if not, then such a signal would be allowed to propagate. If so, then the signal would be terminated. However, the latency of optical-electrical-optical conversion is > 100 ns, thus the spanning tree architecture is not utilized.

Figure 7: Fault Tolerance

2.1.5.3 Expandability Expandability is the third facet of the design approach, wherein the network supports plug-and-play functionality, i.e. the ability to insert and remove nodes from the network and not affect the network's functionality. This is reflected in the spanning-tree architecture, where an additional node is linked to several near by nodes and functionality is preserved. However, due to afore mentioned limitations of the spanning-tree design, a ring topology could be employed. Anywhere in the network a fiber optic line linking a pair of nodes can be cut and a new node inserted.

- 25 -

Figure 8: Expandability 2.2 Detailed Design Detailed design enumerates the basic principals of fault tolerance, expandability, and decentralized control. 2.2.1 Hardware Design Implementation of the hardware network shall involve two distinct steps. The premise of the third step is to organize a plan for gradual increases in sophistication of the design for future design teams. 2.2.1.1 Point to Point Interconnection The simplest implementation of the fiber optic interconnect is to interconnect two cards using a point to point connect, i.e. receiver of a given board is linked to the transmitter of the other board, and vice versa.

Figure 9: Point to Point Interconnect

- 26 -

2.2.1.2 Unidirectional Bus with Single Fiber The second part of the hardware design involved the unidirectional ring shown in the figure below. Each node is comprised of a Xilinx Virtex II Pro board connected with a USB cable to a computer (used as a user interface). The couplers are all 50/50 split couplers from a company named Fiberdyne. Multimode fiber optic cabling was used to interconnect components in the arrangement shown below.

Figure 10: Unidirectional Bus Hardware Design 2.2.1.2.1 Parts List

Table#1: Parts List Component Part Number Manufacturer Split Ratio Coupler F7H7211100 Even Split Fiberdyne 50/50 Splitter F717211100 Even Split Fiberdyne 50/50 One meter cable F9C11B2HGC7701M Fiberdyne … Two meter cable F9C11B2HGC7702M Fiberdyne … 2.2.1.3 Bidirectional Ring with Single Fiber The third part of the design will be the implementation of bidirectional data transfer. One fiber is used as a path between nodes, and a coupler/splitter sub-network with each node is used to couple information onto and off of the fiber optic line.

������

�

�

- 27 -

Figure 11: Bidirectional with single fiber

2.2.1.3.1 Coupler/Splitter Sub-network of Bidirectional Network This sub-network supports bidirectional traffic. The goal of this design is to minimize loss across the sub-network. The splitter for this sub-network will evenly divide power between couplers.

Figure 12: Bidirectional Sub-network

2.2.1.3.1.1 Scattering Parameters The use of scattering parameters quantifies the sub-network in a black box model. Note that the units a and b below are in Watts , and that scattering parameters have no units. All instances of "a" refer to power entering the sub-network and "b" refers to power exiting the sub-network.

- 28 -

The splitters are three port devices. The receiver usage is actually as a coupler; the transmitter uses the splitter as an actual splitter. Below are the scattering parameters for the splitter, based on the Wilkinson Splitter design.

1 1

2 2

3 3

0 0.707 0.707

0.707 0 00.707 0 0

b a

b a

b a

� � � � � �� � � � � �= ⋅� � � � � �� � � � � �� � � � � �

Figure 13: S-Parameters/Splitter

For the transmitter, port 1 is where the transmitted signal enters and is split evenly between ports 2 and 3. For the receiver, port 1 is where signals are received and ports 2 and 3 are where signals from either direction are coupled into receiver. The couplers are four port devices. The design is based on a coupler-coupler configuration with internal termination. Below are the scattering parameters:

1 1

2 2

3 3

4 4

0 0.707 0.707 00.707 0 0 0.7070.707 0 0 0.707

0 0.707 0.707 0

b a

b a

b a

b a

� � � �� �� � � �� �� � � �� �= ⋅� � � �� �� � � �� �

� �� � � �

Figure 14: S-Parameters Coupler

Port 2 is always the outer port (i.e. the port that is directly connected to another node through fiber optic cable), port 1 is always the inner port (i.e. the port that is connected to the other coupler), port 3 is connected to the receiver, and port 4 is connected to the transmitter. Overall, the sub-network is a four port device with the following scattering parameters:

- 29 -

1 1

2 2

3 3

4 4

0 0.5 0.5 00.5 0 0 0.50.5 0 0 0.50 0.5 0.5 0

b a

b a

b a

b a

� � � �� �� � � �� �� � � �� �= ⋅� � � �� �� � � �� �

� �� � � �

Figure 15: S-Parameters Overall

Taking the overall sub-network as a four port device, ports 1 and 2 are where data enters the node from an external source, port 3 is where information is transferred into the receiver, and port 4 is where data is transmitted into the network from the transmitter. 2.2.2 Software Design There are two areas of software for this project: the time division multiplexing (TDM) scheme, and the transmission and reception of packets using the Aurora protocol. 2.2.2.1 Basic TDM Scheme

Figure 16: Basic TDM Scheme In order to use more than two nodes, a basic TDM scheme has been designed to force the nodes to share the available transmission time. While this scheme was never put into use

- 30 -

due to a lack of reliable transmission, this is how to use multiple nodes on a shared transmission medium. Three modes of operation are available for each node: primary controller, secondary controller, and controlled. 2.2.2.1.1 Primary Controller The primary controller manages the network TDM scheme functionality in addition to regular node functionality. Time slots are allocated equally between the nodes, and the primary controller sends a signal at the beginning of each time block indicating which node has permission to transmit. 2.2.2.1.2 Secondary Controller A secondary controller has the same capability as the primary controller, but is not actively sending control signals. When it detects a failure in the primary controller caused by a network break or the hardware failure of the primary node, it assumes the primary controller mode and begins sending control packets. 2.2.2.1.3 Controlled Controlled nodes receive packets all the time, but only transmit for one time slot after receiving a control packet granting them permission to transmit. 2.2.2.2 Advanced TDM Scheme

Figure 17: Advanced TDM Scheme layout

- 31 -

An extension to the basic TDM scheme is an advanced TDM scheme. While not implemented yet, the scheme uses a more advanced algorithm to decide how many time blocks each node gets during a time slot. This scheme requires each node to report how much network use it anticipates during a specific time slot. The controller then calculates how many time blocks each node gets during that time slot proportional to the needs of the other nodes. In order to prevent transmissions from overlapping caused by slight timing discrepancies, a guard time is needed at the end of each time block and a synchronization time at the beginning. This ensures that the control signal will have propagated to every node before the node begins transmission, preventing interference. Guard time is a pause after transmission of a block to counter the estimated variation of each processor’s clock, while synchronization time is used to synchronize clocks before transmission from a new source. 2.2.2.3 Aurora Protocol The Aurora protocol is a very basic 8 to 10 bit encoding scheme used by Xilinx development boards to perform high speed transmissions. While not as robust as protocols such as Ethernet, it provides sufficient error checking for our needs and has a lower processing overhead than other protocols. The Aurora protocol originally sends clocking signals based on a timer, but since the nodes need to only transmit during their time block it has been modified to send the clocking sequence during the synchronization time of each time block. Information about the unmodified Aurora protocol is available at the following web address: http://www.xilinx.com/products/design_resources/conn_central/grouping/aurora.htm The Aurora protocol was implemented by utilizing VHDL code to describe how the FPGA chip on the Xilinx development board should behave. As illustrated in Figure N, there are several components involved in using the Aurora protocol. The hard components in the design are the PowerPC processor and the block RAM (BRAM), the rest is described by VHDL. The PPC is used to run the control program, and packets to be sent and those received are stored in their respective BRAMs. When a packet is to be sent, the bus transceiver transfers the packet from the TX BRAM through the processor local bus (PLB), a fast 64 bit wide bus, to the TX FIFO. Then, when the Aurora core receives a signal through the 32 bit onboard peripheral bus (OPB), it reads from the TX FIFO, encodes the packet into the Aurora format, and transmits it to the network. When a packet is received, it is decoded from the Aurora format and placed into the RX FIFO by the Aurora core. A counter that is accessible in the software via a register is incremented for each packet received, and the bus transceiver can be instructed through software to move the packet into RX BRAM.

- 32 -

����

��������

�

������

�������

������� �������

��������������

�������

������

������������ ��!

Figure 18: Aurora Architecture

2.3 Implementation Process description The implementation process involves programming the NICs to function as network interface cards, connecting nodes through fiber optic cable, couplers, and splitters, and monitoring output at four different nodes. 2.4 Testing of the End Product and its Results Testing can be broken into fiber optic testing, NIC software testing, and system integration. The tools used for testing are itemized first. 2.4.1 Lab Equipment Utilized

• Digital Sampling Oscilloscope: This was used to see what the signal looked like as it propagated through a fiber optic path.

• Pulse Generator: This was used as the trigger source for the oscilloscope and was also used to generate a bit pattern to drive the transceiver model.

• Power Meter: This was used to measure the power level of signals as the signal propagated through the loop.

• Lab computers, DC power supplies, and fiber optic multimode cable. 2.4.2 Software Used

- 33 -

• Xilinx Platform Studio: This is the software used to download both the VHDL and C code that runs on top of the FPGA. This software is primarily available in the dependable computing laboratory.

• TeraTerm Pro: This is used to monitor output of the Xilinx board via an RS-232 port.

2.4.3 Fiber Optic Testing Testing the fiber optic subsystem requires assembling the fiber optic network and putting a signal on at a transmitter port of any particular node. One generates the signal by following these steps:

1) Connect the transceiver to the power supply set to 3.3 volts. 2) There will be a wire coming out of (specific pin); this wire, when grounded,

has the effect of closing ground-open switch inside the transceiver. Closing the switch tells the transmitter to transmit. Do not close this switch at this time.

3) Connect to the differential RF receive ports a bit stream from the function generator (signal to the positive terminal, inverted to the negative (or vice versa)). Connect the trigger output from the function generator to the external trigger input of the oscilloscope. Set the input frequency to 100 MHz.

4) Connect a fiber optic cable to the transmit terminal of the optical transceiver. The mating connector style should be the LC style.

5) Connect the opposite end of this cable to the input of a 2 x 2 coupler, 50/50 split. This should be a transmitter port for the over all network.

6) Connect a patch cable to the receiver port of any given coupler in the network. Connect this to the digital sampling oscilloscope.

7) The oscilloscope should be set to display in dBm’s of power on the y-axis and time on the x-axis.

8) What the experimentalist should see are two signals that are at different power levels: one signal, arriving earlier, should be at a greater power level than the other signal, with should be at lower power level.

2.4.4 Software Testing Testing the software and the operation of the development board requires the user to have a working project for Xilinx Platform Studio and a serial connection to view output from the development board. Since the FPGA chip has two PPC processors, testing began on a single board, transmitting packets between the two processors, and then moved to multiple boards. 2.4.4.1 Basic Communication Between Two Processor Cores Testing the basic communication involved setting up two copies of the Aurora hardware on a single board, with one copy using each PPC processor. The connection between

- 34 -

processor cores is done via a SATA cable with both ends plugged in to the same Xilinx development board, and one processor core controlling each end. To run the test program on the development board, first the VHDL must be synthesized into a bitstream that tells the FPGA chip how to behave. Then, the program that runs on the PPC to control the hardware must be compiled, and finally the whole thing is downloaded to the board. When it is finished downloading, the program runs automatically, and prints (via the serial RS232 port, which is viewed using Tera Term or similar terminal program) both the packets sent and the packets received for comparison. When they match up, the transmission was a success. We achieved successful tests at this level. 2.4.4.2 Basic Communication Between Two Boards Once success was achieved using a single board looping back to itself, the next goal was two boards communicating. Using a similar setup as the single board test, two boards were connected by a SATA cable and the test program was modified to accommodate the new configuration. At this stage in testing, a few errors showed up in the communication, and the rest of the time was spent trying to find and fix all the causes of error. While progress was made in fixing some of them, not all problems were fixed in the time frame we had, so reliable communication was never achieved. 2.4.4.3 Connection Between Boards For simplicity sake, connection between boards was first done through a SATA cable. Using a single cable that easily and quickly connects and disconnects was simpler to get started with. Once the project had some success in transmission, the transition to SMA connections was made. These were more difficult to deal with, since they have to be screwed in when connected and unscrewed when disconnected, but they were necessary for the third step, which was connecting the laser transceivers and fiber optics. Basic tests were successful using the fiber optics to connect two boards, even though the final configuration of fiber optics and nodes were not integrated together. 2.4.5 System Integration Due to the unreliable communication, the two major components of the system, the fiber optic network and the development board nodes were never integrated together. 2.5 End results of the project The end result of the project is the two systems that were developed. If the development board nodes were completed, it should be a relatively trivial task to integrate them into the fiber optic network for a complete, functional network.

- 35 -

3. Resources and Schedules This section lists team resources and schedules for the tasks. 3.1 Resources This section will approximate the resources that will be required in order to successfully complete this project. 3.1.1 Personnel The three tables below indicate the estimated amount of time spent on the following tasks by each group member: Task #1: Problem Definition

a) Determine deliverables b) Define available resources c) Define client requirements

Task #2: Technology considerations and selection a) Understand technological concepts b) Define available technologies c) Technology selection for the project

Task #3: Prototype Design a) Define factors to meet requirements b) Selection of components c) Finalize system design d) Order components

Task #4: Prototype Implementation a) Install hardware and software b) Test functionality

Task #5: Prototype Testing a) Implement tests and gather data b) Compare test data to requirements

Task #6: End Product Documentation a) Demonstration to advisors b) Class demonstration c) Industrial panel demonstration

Task #8: Project documentation and reporting a) Project plan b) Design report c) Project poster d) Final report

- 36 -

Table #2: Original Personnel Effort Requirements (hours) Name David Sheets Adam Fritz Layth Al-Jalil Jay Becker Total

a) 7 6 6 6 25 b) 10 15 12.5 12.5 50 Task #1 c) 12.5 12.5 10 15 50 a) 10 11 12 12 45 b) 10 9 8 8 35 Task #2 c) 9 9 14 13 45 a) 8 6 9 7 30 b) 10 10 10 10 40 c) 10 7 7 6 30

Task #3

d) 8 6 6 5 25 a) 5 5 15 15 40 Task #4 b) 13 16 7 4 40 a) 8 9 10 8 35 Task #5 b) 10 10 10 10 40 a) 1 1 1 1 4 b) 1 1 1 1 4 Task #6 c) 1 1 1 1 4 a) 12 11 9 8 40 b) 8 9 11 12 40 c) 7 6 8 4 25

Task #7

d) 10 8 6 6 30 Total 170.5 168.5 173.5 164.5 677

Table #3: Revised Personnel Effort Requirements (hours)

Name David Sheets Adam Fritz Layth Al-Jalil Jay Becker Total a) 5 3.5 7 3 18.5 b) 12 10 10 5 37 Task #1 c) 13 10 10 8 41 a) 17 15 18 11 61 b) 20 9 12 4 45 Task #2 c) 8 6 10 10 34 a) 10 8 10 8 36 b) 10 12 10 12 44 c) 45 15 40 15 115

Task #3

d) 10 5 10 5 30 a) 12 12 12 12 48 Task #4 b) 14 15 15 15 59 a) 24 25 25 26 100 Task #5 b) 29 30 30 31 120 a) 13 14 14 14 55 b) 8 9 10 8 35 Task #6 c) 22 22 21 24 89 a) 9 5 10 5 29 b) 20 19 20 18 77 c) 11 12 12 12 47

Task #7

d) 39 40 42 40 159 Total 335 309 348 303 1279.5

- 37 -

Table #4: Final Personnel Effort Requirements (up to April 1, 2007)

Name David

Sheets Adam Fritz Layth Al-Jalil Jay

Becker Total

a) 5 3.5 7 3 18.5

b) 12 10 10 5 37

Task #1 c) 13 10 10 8 41

a) 17 15 18 11 61

b) 20 9 12 4 45

Task #2 c) 8 6 10 10 34

a) 10 8 10 8 36

b) 10 12 10 12 44

c) 45 15 40 15 115

Task #3 d) 10 5 10 5 30

a) 15 12 15 18 60

Task #4 b) 14 15 15 15 59

a) 24 20 25 28 97

Task #5 b) 29 30 30 31 120

a) 13 14 13 11 51

b) 8 15 12 8 43

Task #6 c) 22 22 21 24 89

a) 9 5 10 5 29

b) 20 19 20 18 77

c) 11 12 12 12 47

Task #7 d) 39 40 42 40 161

Total 354 297.5 352 291 1294.5

The final personnel effort requirements table given above differs slightly from the previous revised personnel effort requirements table partially due to the prototype implementation of hardware and software. Various technical issues have arisen throughout the implementation of our design which has resulted in additional hours required by multiple team members. Secondly, previous estimations with respect to the presentations given this semester proved to be inaccurate. Additional hours were required in order to deliver the quality of presentation that was expected. 3.1.2 Other Required Resources The three tables below indicate the estimated resource requirements anticipated as well as the final resources required by the team.

- 38 -

Table #5: Original Other Resource Requirements

Equipment and Misc. Resources Item Team Hours Cost

Optical NICs 0 $4,000

Fiber cable 0 $50

2 PCs 0 Donated

Misc. hardware 0 $500 Poster Board 0 $50

Table #6: Revised Other Resource Requirements

Equipment and Misc. Resources Item Team Hours Cost

Optical NIC(s) 0 $5100 Fiber cable 0 $50 2 PCs 0 Donated Misc. hardware (splitters, couplers, etc.) 0 $500 Poster Board 0 $50

Table #7: Final Other Resource Requirements Equipment and Misc. Resources

Item Team Hours Cost Xilinx Virtex II Pro develop boards (NIC) 0 Donated Multimode Fiber cable 0 $250 2 PCs 0 Donated

Misc. hardware (couplers, splitters, etc.) 0 $1,900 Poster Board 0 $25

Our final resource requirements differ from our previous estimates in that due to financial and technical constraints we were unable to purchase 10 gigabit NIC(s). Xilinx Virtex II Pro boards donated by ISU have been implemented into our design in order to continue with a design that is compatible with 10 gigabit communication. Additionally, the hardware required to simulate a four node network required more funds than anticipated. 3.1.3 Project Costs The three tables that follow summarize the total financial costs required by this project and the modifications that have been made to these estimates throughout the project. There are differences in the final financial cost calculations as a result of the decision to use NIC(s) donated by the university rather than purchase from a vendor as well as the unforeseen expenses of our coupler/splitter sub-network design.

- 39 -

Table #8: Original Financial Costs Parts and Materials ITEM W/O Labor With Labor

Optical NICs $4,000.00 $4,000.00 Fiber cable $50.00 $50.00 2 PCs Donated Donated Misc. hardware $500.00 $500.00 Poster Board $50.00 $50.00 Subtotal $4,600.00 $4,600.00

Labor at $11.00 per hour David Sheets $3,894 Adam Fritz $3,272 Layth Al-Jalil $3,872 Jay Becker $3,201 Subtotal $14,239 TOTAL $4,600.00 $18,839.00

Table #9: Revised Financial Costs Parts and Materials ITEM W/O Labor With Labor

Optical NIC(s) $5,100 $5,100 Fiber cable $50 $50 2 PCs Donated Donated

Misc. hardware (splitters, couplers, etc.) $500 $500

Poster Board $50 $50 Subtotal $5,700.00 $5,700.00

Labor at $11.00 per hour David Sheets $3,894 Adam Fritz $3,272 Layth Al-Jalil $3,872 Jay Becker $3,201 Subtotal $14,239 TOTAL $4,600.00 $19,939.00

- 40 -

Table #10: Total Final Financial Costs Parts and Materials ITEM W/O Labor With Labor

Xilinx Virtex II Pro develop boards (NIC) $0.00 $0.00

Multimode Fiber cable $250.00 $250.00 2 PCs $0.00 $0.00

Misc. hardware (couplers, splitters, etc.) $1,900.00 $1,900.00

Poster Board $25.00 $25.00 Subtotal $2,175.00 $2,175.00

Labor at $11.00 per hour

David Sheets $3,894 Adam Fritz $3,272 Layth Al-Jalil $3,872 Jay Becker $3,201 Subtotal $14,239 TOTAL $2,175.00 $16,414.00

3.2 Schedules The following section illustrates using Gantt charts the schedule followed throughout the course of this project. 3.2.1 Project Schedule This section details how long the project has taken based on how long each task took with respect to the order the tasks must be completed in.

- 41 -

Figure 19: Original Project Schedule

Figure 20: Revised Project Schedule

- 42 -

Figure 21: Final Project Schedule

3.2.2 Deliverables This section shows when the project deliverables were due.

Figure 22: Deliverables Schedule

4. Closure Materials This section includes client, faculty advisors, and group member information, as well as a summary of the project and references to resources used in preparing the final report.

- 43 -

4.1 Project evaluation This project has been mostly successful, and although we did not meet the 10Gbps requirement due to hardware deficiencies, we provide methods by which 10Gbps could be attained using better hardware. In this research project a wide variety of information has been gathered and experimentation recorded. The concepts discussed in this paper should prove to be useful ideas in future designs. 4.2 Commercialization As this is a proof of concept project, no commercialization analysis was made by our team. 4.3 Recommendations The principle recommendations for project continuation revolve around four components: signal filtering, the NIC, software implementation, and signal throughput. 4.3.1 Project Continuation The point of terminus for this project was building two fiber optic loops wherein the signal propagated in opposite directions in separate fibers. The reason this design was not chosen is because the available transceivers did not output an electrical signal proportional to the fiber optic input intensity, but rather output a uniform electrical signal regardless of the input intensity; this means that there was no ability to filter two fiber optic signals of similar amplitude. Using two fibers doubles the amount of equipment used and doubles the potential for breaks and faults in the system. Generally speaking, decreasing the amount of equipment decreases the risk of failure. Therefore a principle goal for the next team to engage this project should be to reduce the amount of equipment used and needed. To that end all further recommendations, save increased throughput, follow. 4.3.1.1 Signal Filtering This system DOES NOT employ wavelength division multiplexing, thus only one wavelength of signal propagated through the system. If two signals are to be split and transmitted about a single fiber optic loop, then each receiver in the network will receive an additive signal where the weak signal may be as much as 20 dB (10-2) below the strong signal. Any transceiver used in such a design should either directly convert the additive fiber optic signal to an equivalent RF signal (wherein the additive nature of the received signal is preserved) or, more preferably, a transceiver that locks to the more powerful of the two signals and ignores the additive effects of the weaker signal but in the absence of the strong signal can detect the weak signal.

- 44 -

The preferred method of filtering two signals is in optics as part of the transceiver design. If filtering of two signals is done in the RF domain, there are several important factors to consider:

• Because fiber optics is an OOK system, 10Gbps bit rate means a 10GHz signal. 10GHz circuits are difficult to design, thus early on effort should be made to minimize the amount of work done in the RF domain, and that which needs to be done should be itemized early on. Research should be done for circuitry that operates at 10GHz. The current design utilizes RF technology at 3 GHz.

• The technology needed to design an RF circuit is ADS, an RF design package, Cadence, a digital circuit design package, access to spectrum analyzer to test the bandwidth of the circuit (and to ensure that the design frequency falls within that bandwidth), and access to a function generator to test the operation of the circuit.

4.3.1.2 The NIC Most COTS NICs that support 10Gbps transfer do so over multiple lines, meaning a parallel signal between boards using some duplex mode of operation. Of the 10 Gbps boards available, most of them are geared toward Ethernet. Ethernet was not chosen for this project primarily because Ethernet is designed for a hub and spoke topology. To drive the project toward Ethernet would require having access to the firmware and software programming of a given NIC to redefine the operation of the board; the companies pursued during the initial project run where reticent to allow access to firmware and software for modification; such things are, of course, proprietary. Thus if the project were to move beyond 3 Gbps and where to attempt 10 Gbps transfer, then, due to a lack of 10 Gbps fiber optic transceivers that are affordable, such a group should immediately consider how to design a system that transmits in parallel. The Xilinx Virtex 4 is a board that claims 10 Gbps transfer and should be investigated as a potential 10 Gbps solution. Myrinet manufactures an Ethernet board that supports 10 Gbps Ethernet. 4.3.1.3 Software Implementation The most frustrating element of this project was software, namely getting simple software to run on a Xilinx Virtex II Pro board. The reason the software was difficult to implement primarily stemmed from the use of VHDL code that was not exactly compatible with the board being used, and also there was an incomplete knowledge of our part as to how the software interfaced with the hardware. Because there is no operating system running on the Xilinx boards, C code directly interfaces with memory, which means that an understanding of the hardware is necessary before software development can begin. Any team should begin by building a comprehensive flowchart of how the VHDL modules, and hence the hardware emulated by the FPGA, interacts. This will aid a mental picture of device operation.

- 45 -

4.3.1.4 Signal Throughput The amount of throughput put on the fiber optic line is primarily a function of the fiber optic transceiver being used and the RF transceiver driving it. With the confines a small budget, increasing signal throughput is more a task of adding levels of complexity to the design by adding more lines. To increase throughput in a more graceful manner, new boards must be purchased along with transceivers that can be driven at higher frequencies. 4.4 Lessons Learned Some lessons we learned while completing this project are signal processing issues related to fiber optic communications and that vendors are reluctant to allow public access to product prices, specs, and quality customer service. We also learned a great deal about fiber optics, VHDL, and embedded development pertaining to the Xilinx boards. 4.5 Risk and Risk Management Anticipated risks were primarily that a lack of familiarity with the subject material would slow development, but we successfully mitigated the risk by consulting with graduate students and our advisors, all of whom were very insightful. We ran into one unexpected problem with the successful transmission/reception of packets, and solved it both by initially coding around the problem and eventually by fixing the project to remove the problem. 4.6 Project team information The project client, advisors, and team consist of the following people: Table #11: Project Team Information

Client

Rick Stevens Lockheed Martin Eagan, MN 651-456-3118 rick.c.stevens@lmco.com

- 46 -

Faculty advisors

Dr. Arun Somani Jerry R. Junkins Professor and Chair Department of Electrical and computer engineering Iowa State University 2211CooverHall 50011 Ames, IA TEL: 515-294-0442 arun@iastate.edu Dr. Mani Mina Adjunct Assistant Professor Department of Electrical and computer engineering Iowa State University 341 Durham 50011 Ames, IA TEL: 515-294-3918 mmina@iastate.edu

Technical advisors

Dr. Robert Weber David C. Nicholas Professor Department of Electrical and computer engineering Iowa State University 301 Durham 50011 Ames, IA TEL: 515-294-8723 weber@iastate.edu

Team members

David Sheets (EE/CprE) 8327 Wallace Ames, IA 50010 319-651-3584 djsheets@iastate.edu Adam Fritz (EE) 218 Ash Ave. Ames IA 50014 515-450-1332 afritz@iastate.edu Jay Becker (CprE/ComS) 4733 Toronto St. Apt. 306 Ames IA 50014 515-290-2669 coln.panic@gmail.com Layth Hamid Al-Jalil (CprE) P.O. Box 9097 Ames IA 50014 isuslal@iastate.edu

- 47 -

4.7 Closing Summary The project's main task is to discover what commercially available 10Gbps optical hardware has the best performance, given the chosen topology. The design that has been selected does not achieve 10Gbps performance because the constraints of our budget did not allow us to purchase more than two boards capable of 10Gbps performance, plus equipment was already available to demonstrate performance at 3Gbps. The team has researched the various bus technologies/topologies as well as optical technologies in order to arrive at the current design recommendation. The team mostly focused on the analysis of the design and required equipment. This research project will serve as an information resource for future design teams at ISU and at Lockheed Martin. The team’s approach has been to utilize the knowledge of the group advisors and graduate students to gain a better understanding of the concepts and technologies that are involved with the project. 4.8 References Stallings, William. High-Speed Networks and Internets, second edition. Prentice Hall, Inc. Upper Saddle River, New Jersey. 2002. Mynbaev, Djafar K. and Scheiner, Lowell L. Fiber-Optic Communications Technology. Prentice Hall, Inc. Upper Saddle River, New Jersey. 2001. Weber, Robert J. Introduction to Microwave Circuits. IEEE Press, New York, NY. 2001 Leon-Garcia, Alberto and Widjaja, Indra Communication Networks – Fundamental Concepts and

Key Architectures. Second edition, McGraw-Hill. Oppenheimer, Priscilla and Bardwell, Joseph. Troubleshooting Campus Networks – Practical

Analysis of LAN Protocols. Wiley Publishing, Inc. Norris, Mark. Gigabit Ethernet Technology and Applications. Artech House, Inc., 2003.

- 48 -

4.9 Appendix 4.9.1 Appendix A The software that controls the transmission in the network uses a time division multiplexing (TDM) scheme. This scheme provides each node with a time window consisting of a time slot, or slots, for transmission while the rest of the nodes receive. In addition, the software system interfaces with PCs for code testing and monitoring. The network transmissions of control and data packets use the Aurora protocol.

�����"���������

����#

����

����$����%

&����������

����'��

&����������

����'��&����������

����'��

&����������

����'��

������������� ��

������������� ��

������������� ��

Figure 23: High level TDM illustration

One of the nodes has primary controller (scheduler) functionality where it signals the beginning of each transmission window in the network as shown in Figure 4. Another node has a secondary controller (back up) functionality where it monitors the networks and assumes the role of primary controller if it detects a failure. The rest of the nodes remain in controlled mode all the time. The multiplexing scheme shall allow multiple boards to communicate over the same fiber optic line. This system shall multiplex access to the line so that only one board is communicating over the line at a given time.

- 49 -

The algorithm for the TDM scheme shall require that each NIC be synchronized and shall broadcast the total amount of data a specific computer, housing the NIC, requires to transmit. Each board will be made aware of all other computer's size requests and each NIC shall use the same algorithm to schedule itself. Specifically, for a given time period, each NIC will be allowed to broadcast on the ring for an amount of time proportional to the amount of data that NIC needs to transmit relative to the total amount of data all NICs wish to transmit. Larger time slices will be scheduled first, and every NIC will access the network in a periodic fashion. Reference Section 2.1.2.7.

This pseudocode describes the functionality of the software which controls the TDM scheme on the NIC(s)

Basic TDM

1. Primary controller setup only (Basic) While the system is running

{ Send a control packet to declare the beginning of a new time slot.

Listen to for acknowledgement from cards that they are ready.

While the number of acknowledgements from cards less than the number of cards. Skip after while if needed. {

Listen to for acknowledgement from cards that they are ready. }

While the number of rounds is less than the number of cards do {

Send control packet addressed only to the round associate with the number of card in this loop intimating the beginning of their round. (By default the primary controller has the first round and the secondary controller the last round). Then wait the time period of a round.

} }

2. Secondary controller setup (listening mode) only (Basic) While the system is running

{ Listen to primary controller packets that declaring the beginning of time slots. If there is no control packets for a period of five time slots from the primary controller.

{ Send a control packet to all the cards in the network of switching control to the secondary controller.

- 50 -

Execute primary controller algorithm and stay in that mode. }

} 3. Controlled setup only (Basic)

While the system is running {

Listen to for primary controller packets that declaring the beginning of time slots.

If there a packet indicates that this is my round. {

While still in my round { If I have a packet to transmit

Transmit the packet Else Do nothing

} }

} Advanced TDM

1. primary controller setup only (Advanced) While the system is running

{ Send a control packet to declare the beginning of a new time slot, which also have their allowance of time blocks for each round in this time slot.

Listen to for acknowledgement from cards that they are ready.

While the number of acknowledgements from cards less than the number of cards. Skip after while if needed. {

Listen to for acknowledgement from cards that they are ready. } While the number of rounds is less than the number of cards do {

Send control packet addressing the beginning of a round. (By default the primary controller has the first round and the secondary controller the last round). While all cards not assigned to their time blocks {

Send control packet addressed to individual card declaring their beginning of pre-assigned time block(s).

Wait the assigned time period of that card time block(s).

Listen and save the card control packet of this round that addresses the need of time blocks for the next time slot.

}

- 51 -

} Calculate the time required block(s) allowances for each time rounds for the next time slot based on the saved control packets from each card in the passed rounds of the current time slot.

}

2. Secondary controller setup (listening mode) only (Advanced)

While the system is running

{ Listen to primary controller packets that declaring the beginning of time slots. If there is no control packets for a period of five time slots from the primary controller.

{ Send a control packet to all the cards in the network of switching control to the secondary controller. Execute primary controller algorithm and stay in that mode. }

} 3. Controlled setup only (Advanced) While the system is running

{ Listen to for primary controller packets that declaring the beginning of time slots. While there is still time rounds {

If there is a control packet indicates that the beginning of my time block(s). {

While still in my block(s) time {

If I have a packet to transmit

Transmit the packet Else Do nothing

} } If this was the round that associate with my card number

Send a control packet that indicates what I need next time slot.

Else Do nothing. } }

- 52 -

4.9.2 Appendix B

S Parameter Set Up Of Coupler and Splitter

Using the following equation for power distributed to two outputs of a splitter or coupler:

( )

( )

2 1

2 2

1cos 1 cos(2 )

2

1sin 1 cos(2 )

2

in

in

PP

PP

θ θ

θ θ

= + =

= − =

The original presumption was to have the net drop through the network be 3dB. This meant that each coupler would have a 1.5dB drop through it, relative to ports 1 and 2. Likewise, relative to ports 1 and 3, there would be a 5dB drop. This calculation led to a coupler with � ~= 0.18 �. This was felt to unfeasible so � = �/8 was chosen. Thus:

2 1

2 2

cos 0.8535538

sin 0.1464478

in

in

PP

PP

π

π

= =

= =

1 1

2 2

3 3

0 0.92 0.38

0.92 0 00.38 0 0

b a

b a

b a

� � � � � �� � � � � �= ⋅� � � � � �� � � � � �� � � � � �

- 53 -

4.9.3 Appendix C

10-Gigabit Network Interface Card Recommendation: Company: Myri-10G 10-Gigabit Ethernet Solutions Product Code: 10G-PCIE-8A-R Price: $895.00

(Myri-10G PCI-Express NIC with a 10GBase-R port)

Specs can be located on the web at the following address:

http://www.myri.com/Myri-10G/NIC/10G-PCIE-8A-R.html

NIC Specs(Partial): Myri-10G network port: 10GBase-R, 10+10 Gbit/s data rate, full-duplex. Serial data is carried on a single fiber in each direction at 10.3125 GBaud, 64b/66b-encoded. Depending upon the XFP transceiver plugged into the socket in the PCI faceplate, the port is 10GBase-SR (850nm wavelength, 26-300m on multimode fiber), 10GBase-LR (1310nm wavelength, up to 10km on single-mode fiber), or 10GBase-ER (1550nm wavelength, up to 40km on single-mode fiber). The port can operate with either Ethernet or Myrinet protocols at the Data Link layer. When operating in Ethernet mode, the port supports Ethernet flow control as defined by IEEE 802.3x. The allowed length of the fiber cable depends upon the XFP transceiver and the quality of the fiber, but in Myrinet mode must not exceed 200m due to Myrinet flow control. (See this Brief Guide to Myri-10G PHYs, pdf, 35KB.)

PCI-Express host port: This NIC is a x8 (8 lane) PCI-Express Add-in Card. It is capable of exchanging data with a host computer at up to 2 GBytes/s (250 MBytes/s per lane) data rate in each direction, full-duplex. The port is fully compliant with the PCI-Express Card Electromechanical Specification Rev. 1.1, and with the PCI-Express Base Specification Rev. 1.0a. The circuit-board edge connector of the NIC will fit mechanically in x8 or x16 physical slots in host computers. The NIC auto-negotiates operation in the widest available mode supported by the slot it is plugged into (x8, x4, x2, or x1).

NIC Accessories: (XFP tranciever)-XFPs for 10GBase-R. The IEEE 802.3ae 10-Gigabit Ethernet Standard specifies PHYs with 10.3125-GBaud, 64b/66b-encoded, packet data over serial fiber at three wavelengths: 10GBase-LR (1310nm, single-mode fiber to 10km)

XFP Price: $900.00