5 - 8 January 2009National Radio Science Meetings1 The ALMA Data Transmission System – Digital...

transcript

5 - 8 January 2009 National Radio Science Meetings 1

The ALMA Data Transmission System – Digital Portion

Chris Langley

ALMA Back End Integrated Product Team

The Challenge

Transmit 4 – 12 GHz

Astronomical Data from the Front End (FE) Band

Cartridges to the Correlator using Commercial Off The Shelf equipment wherever possible.

The Challenge

Transmit 4 – 12 GHz

Astronomical Data from the Front End (FE) Band

Cartridges to the Correlator using Commercial Off The Shelf equipment wherever possible.

The Proposal

Convert FE data Digitally and Optically prior to transmission from each of 66 antennas.

The Flaw

COTS, or any other, D/A converters capable of

4 – 12 GHz inputs were not available during R&D.

The Flaw

COTS, or any other, D/A converters capable of

4 – 12 GHz inputs were not available during R&D.

The SolutionSeparate, or Down Convert, the 4 – 12 GHz two

polarity band into eight 2 – 4 GHz basebands prior to data conversion and transmission.

Astronomical Data Down Conversion & Transmission

P0 IFDCP0

Virtual Parallel Bus

10 Gb/s X 12≤ 15 Km

DTX 3 DRX 3

DRX 0BB 0

P1 IFDC

4-12 GHz

2-4 GHz

256 bits @ 125 MHz

Front End

Correlator

Data Transmission System

P0 IFDCP0LSB

Virtual Parallel Bus

10 Gb/s X 12≤ 15 Km

DTX 3 DRX 3

DRX 0BB 0

P1 IFDC

4-12 GHz

2-4 GHz

256 bits @ 125 MHz

Front End

Correlator

Design Considerations (1/2)

• Operate at OC-192 (9.95328Gb/s) optical fiber signaling speed

• Use of time division digital multiplexing to transform the input signaling rate to the channel signaling rate

• Insertion of fill bits to convert input rate to signaling rate

• Insertion of fill bits to convert input rate to signaling rate• Use of time division digital de-multiplexing to transform

the channel signaling rate to the output signaling rate

the channel signaling rate to the output signaling rate• Elimination of un-needed fill bits upon reception

the channel signaling rate to the output signaling rate• Elimination of un-needed fill bits upon reception• Use of three OC-192 channels per 2 GHz baseband to

achieve required capacity

• Low-voltage differential signaling (LVDS)– Fast rise/fall times– Noise resistant

• Multiple FPGA design per channel– More economical than single FPGA– Ball Grid Array package Lots of IO– 625+ MHz input signal capability

• Multiple FPGA design per channel– More economical than single FPGA– Ball Grid Array package– 625+ MHz input signal capability

• Commercial Optical “Half” Transponders– Change from original design– Became economical– Built in mux /demux, clock recovery

• Multiple FPGA design per channel– More economical than single FPGA– Ball Grid Array package– 625+ MHz input signal capability

• Commercial Optical “Half” Transponders– Economical– Built in mux /demux, clock recovery

• Air cooled (flow through) module, RFI shielded (-50 dBm)

Data Frame Organization

6 bits 5 bits 124 bits 16 bits4 bits4 bits

’s o

Data Transmission SystemCloser View

Formatter &Optical

Transmitters

FiberOpticMUX

3-bit Digitizer

FiberOpticDe-

Mutiplexer

DRX 0Optical

Receivers &De-

Formatter

3-bits X 16@ 250 MHz

X4D (+/-)

D (+/-)

2 – 4 GHzP0

2 – 4 GHzP1

1-bit X 10 Gb/s X 12

120 Gb/s

1-bit X 10 Gb/s X 12

3-bits X 32@ 125 MHz

Data Transmitter Module(Digitizer and Formatter, 4 per Antenna)

Data Transmitter Module4 / Antenna

250 MHz, 15.6 psec stepped delay, 0 dBm

48ms, LVDS

4 GHz, 15.6 psec stepped delay, 0 dBm

125 MHz, 0 dBm

From IFP

From DGCKLRU

3 bitA/D

1:16Demux

AMB 0x5X, X = 0, 1, 2, 3From ABM

AMBSI-2

SPI DG

To FOM

B Bits

D Bits

C Bits

Backplane

5.0 VDC

15.0 VDC

Power Harness

-5.2 VDC

3.3 VDC

PWR FILTERSFrom PSAC

48 VDC

From FOMOptical Keep Alive

2.13-3.95 GHz0 +/- 1 dBm

- 10 dB

ITU Wavelength

21, 27, 33, or 39

ITU Wavelength

25, 31, 37, or 43

ITU Wavelength

23, 29, 35 or 41

Digitizer AssemblyUniversity of Bordeaux

Formatter with 3 Optical Transmitting Transponders

Data Receiver Module(De-Formatter with 3 Optical Receiving Transponders)

Data Receiver Module4 / Antenna

AMBSI-2

FPGAAltera EP1S20F780C5 D Bits

RX Transponder with 1:16 DeMux

From FOAD

156.25 MHz

Data (D Bits)

16 bits @ 625 Mb/s

Recovered Data Clk

625 MHz

Steering Clk

FPGAAltera EP1S20F780C5 C Bits

From FOAD

156.25 MHz

Data (C Bits)

16 bits @ 625 Mb/s

Recovered Data Clk

625 MHz

Steering Clk

FPGAAltera EP1S20F780C5 B Bits

From FOAD

156.25 MHz

I2C Bus

Control Bits

Data (B Bits)

16 bits @ 625 Mb/s

Recovered Data Clk

625 MHz

Steering Clk

JTAG ChainSPI

-5.2 VDC

1.8 VDC

3.3 VDC

5.0 VDC

-5.2 VDC

1.8 VDC

3.3 VDC

5.0 VDC

-5.2 VDC

1.8 VDC

3.3 VDC

5.0 VDC

DC Power Regulation1

5.0 VDC 3.3 VDC

48 VDC

JTAG Chain

JTAG ChainFPGA Comm.

FPGA Comm.

1.5 VDC

3.3 VDC

1.5 VDC

3.3 VDC

1.5 VDC

3.3 VDC

From Station Bin Motherboard

125 MHz

10 Gb/s

3.3 VDC

32 bits each @ 125 MHz

Configuration EEPROM Altera EPC16QC100

To / From Correlator Station Bin Motherboard

DTS Modules for 1 Antenna

Digitizer Clock

IRAM, NRAO

Data Transmitters

U of Bordeaux,NRAO

Fiber Optic Multiplexer

Jodrell Bank Observatory

Fiber Optic Amplifier /

Demultiplexer

Jodrell Bank Observatory

Data Receivers

DTS Link Tests - ALMA Antenna to LabChile, 8/2008

Things We’d Do Differently …

• Single FPGA per channel!– FPGA logic timing is difficult– Economics will likely catch up

• Closer interaction between hardware and firmware designers– Each should be the other’s backup

• Invite external expert’s opinions sooner during the design process

• Test Stand– Design and build once assembly form factors are determined

• Communication between remote team members was good, but could have been better– Specify an early DTS design review for the international partners

Acknowledgements

• Robert Freund, Principle Engineer, Arizona Radio Observatory

• Paula Metzner, DTS Product Engineer, Atacama Large Millimeter Array, National Radio Astronomy Observatory

… and the entire DTS teams from North America, the University of Bordeaux, IRAM (Grenoble, FR), and Jodrell Bank Observatory (~Manchester, UK).

R. W. Freund, ALMA Memo 420: Digital Transmission System Signaling Protocol, 2002

R. W. Freund and C. Langley, BE Critical Design Review, 2004.

References

Auxiliary Slides

Data Transmission System Overview The Partners

UB Effort

UK Effort

US Effort

To FOAD#2 - #50

To FIR/Correlator

Fibers

10 GB/s Fibers

~185 Similar CablesAntennas & Sites

Control Building

Cables Fibers

~186 Cables

LO M/C Fibers

Fibers

FRTXT(4)

FOPatchPanel

SpliceRack

FOADAnt #1

DFRRXT(4)

LO, M/C Fibers

FromIFP,2 - 4 GHz,Two

Polarities

2 base bands x 96 bits

@ 125 MHz

Antennas 2- 50

Antenna 1

Positions for 49 more Antennas

120 GB/s Fibers

System Requirements

• Repeatable latency with no loss of samples

System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)

System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s

System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source

System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source• Nyquist sampled data

System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source• Nyquist sampled data• 3-bit data word

System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source• Nyquist sampled data• 3-bit data word• Data transmission synchronized with ALMA

timing

System OverviewExplicit requirements

• 4 GSa/s per 2 GHz bandwidth IF channel

• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample

• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels

• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels• 96 Gb/s per antenna (120 Gb/s encoded data)

• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels• 96 Gb/s per antenna (120 Gb/s encoded data)• 250 MHz input word rate (96-bit wide parallel word)

• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels• 96 Gb/s per antenna (120 Gb/s encoded data)• 250 MHz input word rate (96-bit wide parallel word)• 125 MHz output word rate (192-bit wide parallel

word)• Grouping of a polarization pair: 24 Gb/s per pair

word)• Grouping of a polarization pair: 24 Gb/s per pair• Walsh function 180° switching

word)• Grouping of a polarization pair: 24 Gb/s per pair• Walsh function 180° switching• 15 Km (maximum) distance

System OverviewImplied requirements

• Configurable if not deterministic timing (repeatable latency)

• Fast frame synchronization

• Fast frame synchronization• Continuous transmission of data

• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life

• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life• Operation independent of payload content

• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life• Operation independent of payload content• Testing strategies

• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life• Operation independent of payload content• Testing strategies• Economical implementation

Auxilary Slides

DTS Single Bit Data Path(VHDL Top Level)

Frame Implementation

• 160 bit frame• 128-bit payload• 10-bit synchronization word• 5-bit meta-frame sequence word• 1-bit meta-frame index• 16-bit odd parity check word

Design Decisions

• Use of a data block (frame) to facilitate multiplexing

Design Decisions

• Use of a data block (frame) to facilitate multiplexing• Use of a synchronization (framing) word to facilitate

frame detection

Design Decisions

frame detection• Use of scrambling techniques to minimize bit sequence

effects, low frequency content, and to maintain signal balance

Design Decisions

• Use of a meta-frame to synchronize the reception of multiple channels

Design Decisions

• Use of a meta-frame index to synchronize reception to ALMA timing under varying propagation delays

Design Decisions

• Use of a meta-frame index to synchronize reception to ALMA timing under varying propagation delays

• Use of a checksum word to facilitate continuous monitoring of received data integrity

Auxilary Slides

DTX Formatter FPGAs and Transponders

625 MHz

FIBER10 GBs

TTX FRAME DATA D

TTX FRAME DATA C

TTX FRAME DATA B

TTX SPI

FRAMECLK

DIGITIZER SPI

SYSCLK

DIGITIZER

CLOCK DIVIDER

250MHzphase shifted

Auxilary Slides

DRX De-Formatter FPGAs and Transponders

BB1625 MHz

625 MHz

FIBER10 GBs

TRX MUX’D DATA D

TRX MUX’D DATA C

TRX MUX’D DATA B

TRX 3 TOP_C

EEPROM

TRX I2C

I2CSPI

SYSCLK

REF CLK

CORRELATORDATA

196 BITS @ 125 MHz

Auxiliary Slides

Frame Synchronizations

►Frame synchronization (10-bit synchronization word)

• Multi-channel synchronization (5-bit meta-frame sequence word)

• Determination of propagation delay (1-bit meta-frame index)

Auxilary Slides

• Frame synchronization (10-bit synchronization word)• Multi-channel synchronization (5-bit meta-frame

sequence word)• Determination of propagation delay (1-bit meta-frame

index)

Auxiliary Slides

Frame synchronization (10-bit synchronization word)

• Unique or “unique enough” pattern to minimize acceptance of erroneous patterns in random data

• Long enough pattern to eliminate the acceptance of erroneous pattern in static data

• Partitioned pattern to eliminate the acceptance of a correct pattern in an incorrectly configured system

• Three acceptance stages required to qualify a 10-bit quantity as the synchronization pattern

Auxiliary Slides

Stages for Frame Synchronization

• Search:

selection of an initial location within the serial bit-stream followed by the shifting of the location until a candidate synchronization word is located

• Check:

continued observations in subsequent frames until unsuccessful criterion (failure)

• Monitor:

once confirmed, continuous monitoring of all frames to ensure proper operation

Auxiliary Slides

• Frame synchronization (10-bit synchronization word)

►Multi-channel synchronization (5-bit meta-frame sequence word)

• Determination of propagation delay (1-bit meta-frame index)

Auxiliary Slides

Multi-channel synchronization (5-bit meta-frame sequence word)

• Sequence word large enough to accommodate worst case relative variation in propagation delay across the three channels

• Integer number of meta-frames contained within one 48.000ms timing period

• Transmitter simultaneously writes the identical incrementing sequence number in frames of all three channels

• Receiver compares and re-times the frames from the three channels thus synchronizing the meta-frames

Auxiliary Slides

• Frame synchronization (10-bit synchronization word)

• Multi-channel synchronization (5-bit meta-frame sequence word)

►Determination of propagation delay (1-bit meta-frame index)

Auxiliary Slides

Determination of propagation delay (1-bit meta-frame index)

• Transmitter uniquely identifies the first meta-frame following a 48.000ms timing event

• Monitor and Control system obtains the count of frames received following the local 48.000ms timing event and the detection of the meta-frame index bit

• Monitor and Control system command the receiver to adjust its internal frame delay to a specific relative value

Auxilary Slides

Scrambling

• Modification of source data to accommodate specific characteristics of the communication channel

• Provides adequate timing for the clock and data recovery electronics

• Provides a signal balance for the AC coupled circuits which minimizes threshold errors

• Pattern is easily produced by a maximally length shift register generator

• Sync word is exempt from scrambling• For 149 -> 0, Result <= output pattern (shifted +1) XOR

input pattern

Auxilary Slides

Data Integrity

• Parity computation is easier than CRC• 16-bit parity word over 144 bits• Each parity bit monitors 9 other bits• Permits continuous monitoring of transmission quality

Auxiliary Slides

Self Test Methods

• 10 GHz clock recovery• Frame detection• Multiple channel synchronization• Scrambled pattern exercises• Random Number Generation• Checksum (parity) checks• FFT of pseudo Front End data (gain flatness,

CW beacon)

Formatter Block Diagram

Protocol Encoder

HalfTransponder

Protocol Encoder

HalfTransponder

Protocol Encoder

HalfTransponder

Sixteen10:1 Mux’s

1:4 Demux

250 Mb/s 625 Mb/s62.5 Mb/s 62.5 Mb/s 10 Gb/s(1bit optical)

20.833 Hz

125 MHz

(128 bits) (160 bits) (16bits)

MCEngine

X5MultiplierDC-DC

Converters

Monitor PointA/D’s & Mux’s

(32 bits)

FPGA 1

FPGA 2

FPGA 3

3.3 VDC

From Digitizer Assembly

From DGCK

To FOM

To / From Digitizer Assembly

From MC / PS

To / From MC / PSBoard

-5.2 VDC

15 VDC 625 MHz

Laser Keep Alive

From FOM

Auxilary Slides

Digitizer Clock Module

Auxilary Slides

Digitizer Clock AssemblyIRAM, Grenoble

Auxilary Slides

Fiber Optic MultiplexerJodrell Bank Observatory

Auxilary Slides

Fiber Optic Amplifier / Demultiplexer

Auxilary Slides

Transmitter Module – Internal View

Formatter

Digitizers

Monitor Control &Power Supply

Backplane

Auxilary Slides

DTS Test Stand

• “Golden” Modules– 2 Data Transmitters– 1 Digitizer Clock– 2 Data Receivers

• Support Electronics– PC with LabView

Interface– System timing– Data Receiver

backplane

• Test Stand– Design and build once assembly form factors are determined

5 - 8 January 2009National Radio Science Meetings1 The ALMA Data Transmission System – Digital...

Documents