Hardware: Old, New, Hardware: Old, New, FutureFuture

Dan Werthimer and Casper Dan Werthimer and Casper CollaboratorsCollaborators

Existing Multi-purpose Existing Multi-purpose HardwareHardware• BEE2 (5 2VP70-7, 20 GB, 18 CX4)BEE2 (5 2VP70-7, 20 GB, 18 CX4)

• iBOB (1 2VP50-7, 8 MB, 2 CX4)iBOB (1 2VP50-7, 8 MB, 2 CX4)

• iADC (Atmel/E2V, 2 Gsps or 2*1Gsps, 8bit)iADC (Atmel/E2V, 2 Gsps or 2*1Gsps, 8bit)

• National ADC (3 Gsps, >2.5 GHz bandwidth)National ADC (3 Gsps, >2.5 GHz bandwidth)

(can ping pong for 6 Gsps, can use dual 1.5 Gsps)(can ping pong for 6 Gsps, can use dual 1.5 Gsps)

• DAC Board, Dual DAC Board (10 bits (9 usuable), 1.5 DAC Board, Dual DAC Board (10 bits (9 usuable), 1.5 Gsps)Gsps)

• SERENDIP 5 (2V6000, 2V1000, 4 ADC’s 128 Msps)SERENDIP 5 (2V6000, 2V1000, 4 ADC’s 128 Msps)

• 10 Gbe Switches (HP, Foundry, Fujitsu….)10 Gbe Switches (HP, Foundry, Fujitsu….)

• 10 Gbe cables (Gore….)10 Gbe cables (Gore….)

Compute Module DiagramCompute Module Diagram

138 bits 300MHz DDR 41.4Gb/s

4GB DDR2 DRAM12.8GB/s (400DDR)

100BTEthernet

5 FPGAs2VP70FF1704

FPGAFabric

MG

T

Memory Controller

IB4X/CX4 20Gbps

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

DR

AM

FPGAFabric

MG

T

Memory Controller

FPGAFabric

MG

T

Memory Controller

FPGAFabric

MG

T

Memory Controller

FPGAFabric

MGT

Memory Controller

IB4X/CX4 40Gbps

IB4X/CX4 40Gbps

IB4X/CX4 40Gbps

IB4X/CX4 40Gbps

Multi-Purpose SpectrometerMulti-Purpose Spectrometer – – Low Low BandwidthBandwidth Aaron ParsonsAaron Parsons

XilinxVirtex-II 6000

FPGA

XilinxVirtex-II

1000FPGA

256 MB DRAM

200 MhzADC

Compact PCIBackplane

Software

200 MhzADC

200 MhzADC

200 MhzADC

I

I

Q

Q

Pol. 1

Pol. 2

{

{

200 Aux. I/O

Agilent ADC-FGPA-CX4 board Agilent ADC-FGPA-CX4 board - ??- ??

20 Gsps 8 bit Agilent ADC20 Gsps 8 bit Agilent ADC

XC5VSX240T XC5VSX240T 6 CX4 ports 6 CX4 ports 120 Gbps xaui120 Gbps xaui 100 Gbps infiniband 100 Gbps infiniband 60 Gbps 10 Gbe 60 Gbps 10 Gbe

BEE3 BEE3

• Chen Chang (designer of Bee1, Bee2 Chen Chang (designer of Bee1, Bee2 (PhD))(PhD))

• Microsoft, Celestica, BerkeleyMicrosoft, Celestica, Berkeley

• available from BeeCubeavailable from BeeCube

• not open source not open source

ROACH ROACH (virtex 5) (virtex 5) Reconfiguable Open Architecture Compute Reconfiguable Open Architecture Compute

HardwareHardware• Mini-Roach – Henry Chen, Rick Raffanti• Roach:

Francois Kapp (lead), Alan Langman… -- KAT

George Peck, Mike Revnell, Steve Durand -- NRAO

Henry Chen – UCB CASPER

Hayden So – Univ. Hong Kong

Next gen correlators, beam formers, spectrometers, VLBI

Roach Roach MechanicalMechanical

• 1 U ATX Enclosure $1501 U ATX Enclosure $150

• ATX power supply $50ATX power supply $50

ATA quad digitizer, ATA quad digitizer, packetizer (planned - 2009)packetizer (planned - 2009)

New FPGA BoardsNew FPGA Boards

• MiniRoach (V5 test board, spectrometer)MiniRoach (V5 test board, spectrometer)

• Roach (MeerKat, NRAO, Hong Kong, UCB) Roach (MeerKat, NRAO, Hong Kong, UCB)

(V5, CX4, DRAM, SRAM)(V5, CX4, DRAM, SRAM)

• Bee3 (4*V5, CX4, DRAM, Microsoft, BeeCube)Bee3 (4*V5, CX4, DRAM, Microsoft, BeeCube)

• Roach II – XC5VSX240T, 6*CX4, 2DDR2, 3 Roach II – XC5VSX240T, 6*CX4, 2DDR2, 3 QDR QDR

ADC Boards – CurrentADC Boards – Current

• Atmel: dual 1 Gsps, single 2 Gsps, 8 bit, 1.5 Atmel: dual 1 Gsps, single 2 Gsps, 8 bit, 1.5 GHzGHz

• National: 3 Gsps, 8 bit, 2.5 GHz bandwidthNational: 3 Gsps, 8 bit, 2.5 GHz bandwidth

two can be interleaved to get 6 Gspstwo can be interleaved to get 6 Gsps

can also install dual 1.5 Gsps (ADC08D1500)can also install dual 1.5 Gsps (ADC08D1500)

need Roach or Miniroach to get full sample need Roach or Miniroach to get full sample raterate

• 64 channel ADC board – 12 bit, 65 Msps 64 channel ADC board – 12 bit, 65 Msps (2wide)(2wide)

ADC Boards - PlannedADC Boards - Planned• Quad ADC (200 Msps, 8 bit), Quad ADC (200 Msps, 8 bit), PCB Layed Out, HenryPCB Layed Out, Henry

• MeerKat ADC – MeerKat ADC – single 3 Gsps, or dual 1.5 Gsps, 8bitsingle 3 Gsps, or dual 1.5 Gsps, 8bit

programmable attenuation, gain, sample clock, Oct. 2008programmable attenuation, gain, sample clock, Oct. 2008

• ATA – ADC and FPGA, quad 1.5 Gsps, 8 bitATA – ADC and FPGA, quad 1.5 Gsps, 8 bit

with programmable attenuation, gainwith programmable attenuation, gain

• Agilent 20Gsps, 8 bit (unfunded)Agilent 20Gsps, 8 bit (unfunded)

• JPL optically isolated ADC (Robert Navarro)JPL optically isolated ADC (Robert Navarro)

• E2V ADC ? (5 Gsps, 2 * 2.5 Gsps, 4 * 1.25 Gsps)E2V ADC ? (5 Gsps, 2 * 2.5 Gsps, 4 * 1.25 Gsps)

Misc. BoardsMisc. Boards

• Programmable Attenuator/Gain (RAL, Oren)Programmable Attenuator/Gain (RAL, Oren)

4 GHz BW, -16 to +16 dB in 0.5 db step, Notch Filter4 GHz BW, -16 to +16 dB in 0.5 db step, Notch Filter

(or minicircuits, $50, programable (or minicircuits, $50, programable atten/gain)atten/gain)

• 1 PPS distribution (RAL- Oren Milgrome)1 PPS distribution (RAL- Oren Milgrome)

• VSI connector (converts ZDOC to VSI)VSI connector (converts ZDOC to VSI)

• DAC board, 9 bits, ~1.5 GspsDAC board, 9 bits, ~1.5 Gsps

• Dual Dac Board, 9 bits, ~1.5 GspsDual Dac Board, 9 bits, ~1.5 Gsps

Board Interconnect - Board Interconnect - UpgradableUpgradable

• Problem: Backplanes are short livedProblem: Backplanes are short lived

(S100, Multibus, VME, ISA, EISA, PCI, PCIx, (S100, Multibus, VME, ISA, EISA, PCI, PCIx, PCIe, compactPCI, compactPCIe, PCIe, compactPCI, compactPCIe, ATCA…)ATCA…)

• Solution: Use 10Gbit EthernetSolution: Use 10Gbit Ethernet

(10Gbe, Infiniband, Myrinet, Xaui, Aurora)(10Gbe, Infiniband, Myrinet, Xaui, Aurora)

Copper CX4 (40 meters max) or Optical Copper CX4 (40 meters max) or Optical

CX4 cablesCX4 cables

• Gore CX4 10Gbe cables (1, 2, 3, 15 meter…)Gore CX4 10Gbe cables (1, 2, 3, 15 meter…)

about $75 eachabout $75 each

(not infiniband cables)(not infiniband cables)

(see cable length memo for V2Pro (see cable length memo for V2Pro

1-3 Meter max)1-3 Meter max)

Active CX4 Cables – 45 meter maxActive CX4 Cables – 45 meter max

Optical CX4 Cables – Matt Dexter tests ~$300Optical CX4 Cables – Matt Dexter tests ~$300

10 Gbit Ethernet Switches, 10 Gbit Ethernet Switches, NIC’sNIC’s• HP – Procurve 6 to 8 ports CX4, 24-48 HP – Procurve 6 to 8 ports CX4, 24-48

1Gbe1Gbe

• Foundry – 16 XFP portsFoundry – 16 XFP ports

• Fujitsu – XG700 (12 CX4 ports) Fujitsu – XG700 (12 CX4 ports)

• Fujitsu – XG2000C (20 ports, 16 CX4, 4 Fujitsu – XG2000C (20 ports, 16 CX4, 4 XFP)XFP)

• Myricom – 10 Gbe NIC boards (PCIe)Myricom – 10 Gbe NIC boards (PCIe)

Where to get tested boardsWhere to get tested boards• Build and test them yourself (cheapest for large Q))Build and test them yourself (cheapest for large Q))

• iBOB, iADC, 5 volt cables, Digicom ElectronicsiBOB, iADC, 5 volt cables, Digicom Electronics

Mo Ohady, mo@digicom.orgMo Ohady, mo@digicom.org

• Bee2, enclosure and power supply, SAE MaterialsBee2, enclosure and power supply, SAE Materials

Mario Salazar, Mario Salazar, marios@saemtl.com

coordinate with Ken Lutz, BWRC, coordinate with Ken Lutz, BWRC, lutz@eecs.berkeley.edulutz@eecs.berkeley.edu

• iBOB mounting plates, front panels –iBOB mounting plates, front panels – make or buy – make or buy –

NRAO?NRAO?

• Other Vendors?? Rakesh Mehta, MTE India? Other Vendors?? Rakesh Mehta, MTE India?

Logistics – PresentationsLogistics – Presentations

• Get your PPT, .PDF to Jason ManleyGet your PPT, .PDF to Jason Manley

• Install on Presentation LaptopInstall on Presentation Laptop

• DO NOT USE YOUR OWN LAPTOPDO NOT USE YOUR OWN LAPTOP

• Please wear microphone (see Jason)Please wear microphone (see Jason)

• We are posting video and ppt to web We are posting video and ppt to web sitesite

• Let Jason know if you don’t want your Let Jason know if you don’t want your presentation posted presentation posted

Logistics – LocationLogistics – Location

• Saturday,Sunday at BWRC,Saturday,Sunday at BWRC,

BWRC doors locked – buzz or callBWRC doors locked – buzz or call

• Sunday Night Pizza/Beer at Dan’s – all Sunday Night Pizza/Beer at Dan’s – all invitedinvited

• Monday, Tuesday at Space Sciences LabMonday, Tuesday at Space Sciences Lab

Take Bus to SSL: $1, 12 mins, see program Take Bus to SSL: $1, 12 mins, see program

Monday Tuesday Lunch will be providedMonday Tuesday Lunch will be provided

• Tours of BWRC lab, SSL in the afternoonsTours of BWRC lab, SSL in the afternoons

Logistics: Suggested Logistics: Suggested DonationDonation

$ 50 ( $ 25 for students )$ 50 ( $ 25 for students )

covers morning refreshments, two covers morning refreshments, two lunches…lunches…

$ $ Andrew Siemion Andrew Siemion

Casper Advisory BoardCasper Advisory Board Matthew Bailes, John Ford, Yashwant Gupta,Matthew Bailes, John Ford, Yashwant Gupta,

Glen Jones, Alan Langman, Jonathan WeintroubGlen Jones, Alan Langman, Jonathan Weintroub

Brainstorming Session – Open Mic – Tuesday AMBrainstorming Session – Open Mic – Tuesday AM

Constructive CriticismConstructive Criticism

Ideas for Future Directions, Long Term StrategyIdeas for Future Directions, Long Term Strategy

How best to collaborateHow best to collaborate

Organizational StructureOrganizational Structure

Anything else Anything else REPORT REPORT

Thanks to Workshop Organizers Thanks to Workshop Organizers

Henry Chen, Terry Filiba,Henry Chen, Terry Filiba,

Griffin Foster, Suraj Gowda, Griffin Foster, Suraj Gowda,

Jason Manley, Billy Mallard,Jason Manley, Billy Mallard,

Peter McMahon, Andrew Siemion,Peter McMahon, Andrew Siemion,

Laura Spitler, Mark WagnerLaura Spitler, Mark Wagner