eLBA – Current Developments
Chris Phillips
eVLBI Project Scientist
16 June 2008
ATCA
MopraParkes
Hobart
Ceduna
Tidbinbilla
Sydney
ASKAP
The LBA
LBADR – LBA Data Recorder
• Cousin of MRO/EVN-PC• Commodity PC with VSIB input card• Primarily record onto Apple Xserve RAID
• Control software highly modified from original MRO
• Mark5b emulation mode• eVLBI with TCP or UDP• Very flexible
• Data written to normal Linux filesystem• Realtime sampler statistics• Flexible realtime fringe checking
CSIRO. eVLBI-Aus
LBA status
• Using DiFX software correlator for all LBA operations
• All disk correlation has been done at Swinburne University of Technology (Melbourne)
• eVLBI correlated at Parkes• Disk correlation will transfer to Curtin University (Perth) by August 2008
CSIRO. eVLBI-Aus
DiFX
• All LBA correlation now runs on DiFX• Distributed FX
• MPI parallelization on Beowulf style cluster• Written by Adam Deller at Swinburne University of Technology
• Active development also from Walter Brisken• Available free of charge for scientific research
• Supports LBADR, Mark5a, Mark5b, Mark5c• Crucial part of VLBA sensitivity upgrade• Supports mixed mode correlation• Easy to add support for new formats
CSIRO. eVLBI-Aus
DiFX: Architecture
• Based on MPI• Written in C++• Uses IPP libraries extensively
• 2 types of processes• I/O processes (DataStream)• Compute processes (Core)
• Incoming data stream is time divided and copied round robin from DataStream to Core
• Asynchronous sends mitigate swarming effect
• Supports disk and eVLBI operation• TCP & UDP, filesystem and native Mark5
CSIRO. eVLBI-Aus
APSR Cluster
• ATNF Parkes Swinburne Recorder• Backend for Pulsar processing at Parkes
• 18 node Dell cluster• Dual processor quad core
• VLBI front end I/O cluster PAMHELA• 5 node, dual processor dual core
• 4 (Intel) NICs for network flexibility• Job control from PAMHELA, APSR used as slave
CSIRO. eVLBI-Aus
Image Credit: Shaun Amy
PAMHELA APSRT
eles
cope
Con
nect
ions
Case Study: How not to implement a cluster
• PAMHELA running 32bit Debian Etch• APSR running 64bit CentOS• No shared filesystem • MPI/DiFX does not like mixed 32 and 64bit binaries
• DataStream sending 4 times more data than receiving
• Many issues getting MPI• Mpich would not scale to use full cluster• Openmpi seemed to get confused by multiple
interfaces
CSIRO. eVLBI-Aus
ATNF Observatory Network
• Provided by AARNet• Pair of dark fiber, carrying multiple 10 Gbps SDH
• 2 x 1 Gbps connection to each observatory• Used for commodity traffic (web, email, ftp, ssh, observing etc) and eVLBI
• Each link effectively limited to 512 Mbps for eVLBI traffic
• Sydney to observatory can be run at line rate using bog standard PCs
• 989 Mbps user data (evlbi software)• 20 Mbps typical for scp
CSIRO. eVLBI-Aus
ATCA
Mopra
Parkes
Sydney
To Swinburne
To Seattle 10 Gbps
10 Gbps AARNET3To Perth
Current eVLBI status
• eVLBI being offered via normal “call for proposals” and actively encouraged
• 3x512 Mbps ATCA-Parkes-Mopra• 2x1 Gbps ATCA-Parkes (or Parkes-Mopra)
• Implemented as 2x 2x512 Mbps
• Exclusively layer 2 • Hand crafted traffic engineering for “routing”,
using multiple VLANS
• Connection to Hobart problematic• Effectively single 155 Mbps shared IP network
• Cannot sustain 64 Mbps TCP reliably
CSIRO. eVLBI-Aus
http://noc.atnf.csiro.au/
http://noc.atnf.csiro.au
Network Upgrades
• 1 Gbps link to Swinburne imminent…• Multi-gigabit access to 10 Gbps AARNet3 backbone in progress
• Implementing VPLS “Layer2 over Layer3”• eVLBI to Curtin• New cluster for ATCA
• 3x1 Gbps using distributed approach
• Still waiting on upgraded bandwidth to Hobart
CSIRO. eVLBI-Aus
EXPRES-Oz
• AARNet and ATNF Partners on EXPReS project
• Demonstrate realtime eVLBI from Australia to JIVE
• AARNet provided 1 Gbps light paths from Parkes, Mopra and ATCA to JIVE
• CENIC, Pacific Wave, CANARIE, SurfNet
• Science observation of SN1987a at 1.6 GHz• 11hr observation at 512 Mbps sustained• Using Mark5b UDP• 340msec RTT
CSIRO. eVLBI-Aus
Image Credit: Paul Boven,
Image created by Paul BovenSatellite image: Blue Marble Next Generation, courtesy of NASA Visibible Earth
Contours ATCA 9 GHz super resolved image (0.4” FWHM)
e-APT Demo
• Connecting Shanghai and Kashima to Parkes at 512 Mbps
• AARNet have provisioned 3x 622 Mbps circuits
• Mixture of SDH and layer 2/3• AARNet, CENIC, JGN2, CSTNet, Pacific Wave,
HKOEP• Dedicated Mopra-Parkes link• Presented as gigabit Ethernet
CSIRO. eVLBI-Aus
e-APT Demo: Implementation
• ATNF telescopes using LBADR data format and TCP
• Kashima and Shanghai using JIVE UDP format
• Raw Mark5 data stream with 64bit sequence #
• Shanghai using jive5a control on Mark5a• Kashima written realtime Mark5b converter and also generate JIVE UDP packets
• 1500 MTU used exclusively
CSIRO. eVLBI-Aus
Image created by Paul BovenSatellite image: Blue Marble Next Generation, courtesy of NASA Visibible Earth
CSIRO. eVLBI-Aus
Internet2: Wave of the Future
• Winners of inaugural award!• 10 Gbps connection on Internet2 (continental USA) for 1 year
• Possible usage:• eVLBI to Haystack• eVLBI to JIVE
• Details still being worked through
CSIRO. eVLBI-Aus
Contact UsPhone: 1300 363 400 or +61 3 9545 2176
Email: [email protected] Web: www.csiro.au
Thank you
ATNFChris PhillipseVLBI Project Scientist
Phone: +61 2 93724608Email: [email protected]: www.atnf.csiro.au/vlbi