1

US

Peripheral Crate VMEbus Controller

Ben Bylsma

EMU – ESR

CERN, November 2003

2

B. Bylsma EMU-ESR Meeting, CERN 6.11.03 2

Baseline Crate ControllerBaseline Crate ControllerBaseline Crate Controller

Dynatem D360Ø 25 MHz 68360 based VMEbus Single

Board ControllerØ Operating System: µCLinuxØ Communicates with Linux PC (in USC55)

via 10BaseFL Ethernet (10BaseT has magnetic components)

Ø Software for PC-D360-VME communications has been written and tested.

Ø Radiation tolerant§ Tested with 63 MeV protons.§ Observed 35 upsets with a proton fluence of 0.74x1011 p/cm2.§ Recover from upsets with “system reset”. Requires 47 resets in 10 LHC-years.

Ø Speed§ Transfer rate: 10 Mb/s bit rate, though-put ~ 2.5 Mbits/s achievable.§ Speed of loading FPGA firmware to EPROM dominated by required erase/program

delays on chip.Ø Cost ~ $1100 per unit

Dynatem D360

25 MHz 68360 based VMEbus Single Board Controller (3U VME card supporting A24 D16 transfers)

Operating System: µCLinux (burned on a Flash PROM)

Communicates with Linux PC (in USC55) via 10BaseFL Ethernet (10BaseT has magnetic components) (Dynatem comes with RJ45 connector for 10BaseT, but does not work in a magnetic field. We removed the RJ45 and replaced it with our own 10BaseFL adapter board). A commercially available 10BaseT to 10BaseFL box is used on the PC end.

Software for PC-D360-VME communications has been written and tested. (TCP sockets opened on both ends, Dynatem holds code to translate the commands in packets to VME addresses).

Radiation tolerant

Tested with 63 MeV protons. (At UCD cyclotron)

Observed 35 upsets in a fluence of 0.74x1011 p/cm2.

Recover from upsets with “system reset”. Requires 47 resets in 10 LHC-years. (Probably more frequent reboots will be required due to other causes).

Speed

Transfer rate 10 Mb/s bit rate, through-put ~ 2.5 Mbits/s achievable. (This includes Ethernet overhead).

Speed of loading FPGA firmware to EPROM dominated by required erase/program delays on chip.

Cost ~ $1100 per unit (Cheap compared to most SBC’s)

Using the Dynatem as the EMU peripheral crate controller is a proven solution. Its available, relatively cheap, and the required software has already been developed. The bottom line is, it will work. However this does not necessarily mean that everyone is satisfied with the characteristics of the Dynatem solution.

3

B. Bylsma EMU-ESR Meeting, CERN 6.11.03 3

A New Crate Controller in DevelopmentA New Crate Controller in DevelopmentA New Crate Controller in Development

A VMEbus Controller with Gigabit Ethernet Ø A custom board designed and developed at OSUØ Based on XILINX Virtex-II Pro§ XC2VP4 – FF672 FPGA§ Built in RocketIOTM

• Supports Gigabit Ethernet• 8B/10B Encoding, SerDes, Clock Recovery included (no Logic Cells used)

Ø Optical transceiver (for Gigabit Ethernet)Ø VME Interface BuffersØ Command FIFO (for buffering VME commands)Ø Flash Memory (for storing configurations)Ø Flash PROM (for storing the firmware)

A VMEbus Controller with Gigabit Ethernet

A custom board designed and developed at OSUBased on XILINX Virtex-II Pro

Specifically the XC2VP4 FPGA – FF672 (672 pin FlipChip package) Why the Virtex II Pro?Built in RocketIOTM

Supports several standards including Gigabit Ethernet8B/10B Encoding, SerDes, Clock Recovery included (no Logic Cells used)

Only external component needed for Gigabit Ethernet is an Optical transceiver

Other components on the controller are:VME Interface BuffersCommand FIFO (for buffering VME commands)Flash Memory (for storing configurations)Flash PROM (for storing the firmware)

So why are we developing a new controller? There are advantages over the Dynatem.

4

B. Bylsma EMU-ESR Meeting, CERN 6.11.03 4

AdvantagesAdvantagesAdvantages

General AdvantagesØ Communicates with stand-alone PC (in USC55) via EthernetØ GbE network cards commercially available (with optical fiber

connections).Ø Customizable firmware.

Advantages Over DynatemØ A much simpler crate controller.Ø No 10BaseT to 10BaseFL converters needed.Ø No operating system or software on the controller.Ø No sockets to open for each connection.Ø All programming done on PC.Ø Customized packets reduce overhead (explained on next slide) Ø Faster: transfer rate 1 Gb/s bit rate (compare to 10 Mb/s)

through-put ~ 600 Mb/s (compare to 2.5 Mb/s)Ø Less expensive: ~ $600 per unit (compare to $1,100 per unit)

General Advantages

Communicates with stand-alone PC (in USC55) via Ethernet

GbE network cards commercially available (with optical fiber connections).

Customizable firmware.

Dynatem has software which is customized.

Advantages Over Dynatem

A much simpler crate controller.

From the point of view of user interaction with the controller.

No 10BaseT to 10BaseFL converters needed.

The usual GbE is over fiber, but can be over copper.

No operating system or software on the controller.

Only firmware which is stored on a PROM and loaded in 10’s of milliseconds on power up or reset.

No sockets to open for each connection.

Driver communicates directly to network card

All programming done on PC.

Code is in fewer modules

Customized packets reduce overhead (explained on next slide)

Faster: transfer rate ~ 1 Gb/s (compare to 2.5 Mb/s)

Actual bit rate is 1.25 Gb/s because 8 bits are encoded into 10 bits.

Less expensive: ~ $600 per unit (compare to $1,100 per unit)

5

B. Bylsma EMU-ESR Meeting, CERN 6.11.03 5

AdvantagesAdvantagesAdvantages

Customized Packets (with GbE)Ø Can eliminate TCP/IP layerØ Can utilize jumbo packets (9000 bytes of data)Ø Bit rate is 100 times faster than 10BaseT

Packet Structure in use with Dynatem (with TCP/IP):

Customized Packets (with GbE)

Can eliminate TCP/IP layer

Can utilize jumbo packets (9000 bytes)

You have the potential for packing 1283 (A24 D16) VME commandsin a single packet à 8995/(9000 +38)=99.5%

Bit rate is 100 times faster than 10BaseT

6

B. Bylsma EMU-ESR Meeting, CERN 6.11.03 6

StatusStatusStatus

Hardware DevelopmentØ Non-Restrictive board design.Ø Sent to PC house last week.Ø Most parts are on hand.Ø Board Stuffing to take place

next week.

Firmware DevelopmentØ Will be developed in stages.Ø RocketIOTM implementation

has been previously tried and tested.

Ø Start with rudimentary A24 D16 VME master.

Ø Add other functionality, flexibility, configuration control, and robustness as progress is made.

Hardware Development

Non-Restrictive board design.

Even though we may not utilize or implement some VME64x capabilities, the board is designed to be compatible with almost all VME64x functionality.

Sent to PC house last week.

Most parts are on hand.

Board Stuffing to take place next week.

Firmware Development

Will be developed in stages.

RocketIOTM implementation has been previously tried and tested.

Start with rudimentary A24 D16 VME master.

This means unpacking GbE, buffering to the FIFO, transfer from FIFO to VME port, conform to timing requirements, fill a buffer with any read data, create packet,

and send to GbE.

Add other functionality, flexibility, configuration control, and robustness as progress is made.

This includes other transfer modes, define command structure or custom protocol structure for local and VME commands, add mechanism for setting and saving particular configurations, saving a default configuration, develop handshaking to deal with reset or full buffers, add status and monitoring and error reporting, add modules for system timer, arbiter, interrupt handler, etc.

7

B. Bylsma EMU-ESR Meeting, CERN 6.11.03 7

Any Questions?

9

B. Bylsma EMU-ESR Meeting, CERN 6.11.03 9

Distributing GbE to 60 CratesDistributing GbE to 60 CratesDistributing GbE to 60 Crates

Need a Level 2 GbE Network SwitchØ Level 2 means MAC address level.Ø Must have optical fiber outputs.

Commercially availableØ Netgear makes one with 12 GbE fiber optic

outputs for about $2000.

This slide is not intended to be shown unless questions arise about distributing commands to 60 crates.