DiFX: Software Correlation at
Swinburne for the LBA
DiFX: Software Correlation at
Swinburne for the LBA Adam Deller
Swinburne University/CSIRO Australia Telescope National Facility
Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)
9/6/2007 VSOP-2 meeting
Outline
• History• Correlator architecture• Capabilities• Computing environment and resources
• Usage inside and outside LBA• Future improvements
9/6/2007 VSOP-2 meeting
Correlator History• Commenced with my PhD thesis (2005)• Preliminary work (XF) undertaken in 2004 by Craig West
• First science late 2005• Conversion to distributed system 2006
• Verification with LBA and VLBA correlators, code released late 2006
http://astronomy.swin.edu.au/~adeller/software/difx/
9/6/2007 VSOP-2 meeting
Correlator Architecture
Written in C++, MPI for message passing, Intel IPP for vector arithmetic
9/6/2007 VSOP-2 meeting
Capabilities• Supports LBA, Mk5* (K5->LBA converter)• Pulsar gating, binning or matched filter
• Arbitrary time/frequency resolution• Presently, visibilities in RPFITS format
• eVLBI: data can be streamed from network socket directly into correlator
• Also used for real-time fringe checks*export to linux filesystem
9/6/2007 VSOP-2 meeting
The Swinburne supercomputer• Swinburne supercomputer presently consists of ~300 P4 processors
• Typically “reserve” ~30 processors for VLBI after observing run
• Next month: complete replacement of cluster with ~140 dual CPU, quad core nodes, >1100 cores total
9/6/2007 VSOP-2 meeting
Computing requirements
• DiFX requires little RAM (except extreme cases): processor speed, cache and SIMD support essential
• Performance scales linearly with aggregate bandwidth, near-linear with number of stations
• Realtime example: Max sensitivity LBA (6 x 1 Gbps) requires ~25 new nodes
9/6/2007 VSOP-2 meeting
Typical LBA usage
• Required to correlate ~5 days of 256 Mbps before next session (2 months)
• For each experiment, automatically select and correlate fringe-finders, solve clocks, followed by full correlation.
• Using ~30 (old) machines, high data rate experiments take 2x observe time
9/6/2007 VSOP-2 meeting
Non-LBA applications
• New geodetic array in Australia/NZ to use software correlator: modifications to control, output format
• VLBA is trialling software correlator to run in parallel with hardware correlator: experiments with specific requirements, testing high data rate
9/6/2007 VSOP-2 meeting
Future improvements
• Current version is production: future improvements will be incremental
• Direct read from MkV• Alternative output formats: AIPS++ MS, FITS-IDI, UVFITS
• Support in AIPS: tasks like CVEL currently confuse with LBA S2 (XF)
9/6/2007 VSOP-2 meeting
9/6/2007 VSOP-2 meeting
Interferometry & Correlators
RA
Dec
Interferometry: delay signals from two dishes to common reference and multiply
Common signal, independent noise: average improves S/N
Earth rotates, interferometer samples visibility: Fourier transform of sky brightness
9/6/2007 VSOP-2 meeting
Interferometry & Correlators
u ()
v ()
Finite bandwidth, not monochromatic, therefore visibility varies across band
To get v(), XF correlator accumulates lags, then FFTs
FX correlator FFTs segments of baseband data, and cross-multiply/accumulates
9/6/2007 VSOP-2 meeting
Hardware vs software• Software correlator: program running on a computer cluster/supercomputer
• Hardware correlator: ASIC boards, specialised data transport
• Software is unclocked, could be faster or slower than real-time
• No channel/integration time restrictions
• Floating pt vs int calculations
9/6/2007 VSOP-2 meeting
Why software?• Flexibility - you can do things that are impossible with a hardware correlator
• Rapid (and cheap) development• Add-ons MUCH easier in software • Compatibility• Expandability• For me: Allow disk-based correlation, and improved pulsar binning (sensitivity)
9/6/2007 VSOP-2 meeting
DiFX (Distributed FX)
Master Node
Core 1DataStream 1
DataStream 2
DataStream N
Core 2
Core M
… …
Timerange, destination
Baseband data
Visibilities
Source dataSource data
MPI is used for inter-process communications
9/6/2007 VSOP-2 meeting
DiFX (Distributed FX)
• Configured by text files (like jobscripts)
• Delay modelling - CALC 9 (separate)• Output: RPFITS (built on-the-fly)• Arbitrary time/frequency resolution• Arbitrary pulsar binning (incoherent dedispersion) - allows weighted bin sum
• Real time LBA @ 1 Gbps: 100-200 CPUs
9/6/2007 VSOP-2 meeting
DiFX (Distributed FX)
• Verification: recent successful comparisons with LBA and VLBA
9/6/2007 VSOP-2 meeting
LBA Science
• All require one or more of flexibility, high time/frequency resolution, or sensitivity• Wide field VLBI (Lenc & Tingay)• Masers (Horiuchi)• CDF radio counterparts (Norris et al.)
• RRATs (Kramer et al.)• eVLBI (Phillips et al.)• Pulsar parallax (me)
9/6/2007 VSOP-2 meeting
Worldwide science• So far, motivated by very high frequency resolution, or flexibility and minimum effort for new system• Pulsar scintillation (Brisken) requires extreme frequency resolution (244 Hz channels over 32 MHz bandwidth)
• Geodesy (MPIfR, Bonn, Germany)
• Geodesy, new Australian array (NCRIS)
t
9/6/2007 VSOP-2 meeting
Status: Correlator code• “Correlation” code completed and verified - now finishing GUI and packaging for public release
• VLBA, MPIfR and others keen to continue trialling code once released
• Should be online within a month• PASP paper submitted simultaneously
9/6/2007 VSOP-2 meeting
Status: Observing and reduction• Four sessions of eight allocated, two observed - applications for 2007/2008 year (four more sessions) soon
• Both observed sessions correlated and verified
• Starting to work on atmospheric/ ionospheric compensation and pipeline, crucial for efficient, accurate astrometry
9/6/2007 VSOP-2 meeting
Roadmap to submission
1 month DiFX available online
3 months First four sessions complete
6 months Reduction techniques tested, pipeline started
12 months Pipeline complete, draft begun
18 months Data collection, reduction and thesis draft complete
20 months Thesis submission
9/6/2007 VSOP-2 meeting
Conclusions• DiFX is a general purpose software correlator, publicly available soon
• Already used with success in Australia by the wider VLBI community, and generating interest internationally
• Pulsar parallax program is underway, with the bulk of observations to come
• Transitioning from code to science
9/6/2007 VSOP-2 meeting
9/6/2007 VSOP-2 meeting
Interferometry & Correlators• Major operations performed by FX software correlator:• Delay• Unpack quantized data to float• Fringe rotate• FFT• Correct fractional sample error• Cross multiply and accumulate A B
Datastream
Per station N
Per station N2