Post on 21-Jan-2016
transcript
EURO-VO: GRID and VO
Lofar Information System DesignOmegaCEN
Kapteyn Institute TARGET- Computing Center
University Groningen
Garching, 10 April 2008
Lofar Information System DesignOmegaCEN
Kapteyn Institute TARGET- Computing Center
University Groningen
Garching, 10 April 2008
Edwin A. ValentijnEdwin A. Valentijn
EURO-VO DCA GRID and the VOEURO-VO DCA
GRID and the VO
EURO-VO: GRID and VO
LOFAR @ (20-)30-80 MHz and 120-240 MHz
LOFAR @ (20-)30-80 MHz and 120-240 MHz
- EoR
- transients
- Survey
- astroparticles
- EoR
- transients
- Survey
- astroparticles
EURO-VO: GRID and VO
EURO-VO: GRID and VO
E-Lofar: from experiment to observatory
E-Lofar: from experiment to observatory
EURO-VO: GRID and VO
LOFAR in shortLOFAR in short• Large distributed radio telescope
– In NL 2008: initial roll out 20 stations – In NL 2009: 18 central (24 Gbps) + 18 remote (2 Gbps)
phased array antenna stations each ~4 soccerfields size– Full scale aperture synthesis array, extends 100 km
• Two main bands– High Band ~ 7,500 tiles 120 – 240 MHz – Low Band ~ 7,500 dipoles 20 – 80 MHz
• Digital Radio 40 MHz processing bandwidth– Extreme tunability in time/space/frequency– Large instantaneous sky coverage
• Fibre network, Software Correlator• Run as a broad common-user observatory
– Making the transition now from “project” to “observatory”
• Large distributed radio telescope– In NL 2008: initial roll out 20 stations – In NL 2009: 18 central (24 Gbps) + 18 remote (2 Gbps)
phased array antenna stations each ~4 soccerfields size– Full scale aperture synthesis array, extends 100 km
• Two main bands– High Band ~ 7,500 tiles 120 – 240 MHz – Low Band ~ 7,500 dipoles 20 – 80 MHz
• Digital Radio 40 MHz processing bandwidth– Extreme tunability in time/space/frequency– Large instantaneous sky coverage
• Fibre network, Software Correlator• Run as a broad common-user observatory
– Making the transition now from “project” to “observatory”
E-LOFAR ParticipantsE-LOFAR Participants
• GLOW:-- Effelsberg station delivered-- Garching, Potsdam, Tautenburg stations ordered--Jülich station federal funding proposal submitted
• LOFAR-UK:– Chilbolton, Cambridge, Jodrell, Edinburgh– 1 station funded, site to be chosen--3 more stations
• Sweden:– Onsala station funded
• + Poland, Italy, Austria, Bulgaria, Ireland, France
• GLOW:-- Effelsberg station delivered-- Garching, Potsdam, Tautenburg stations ordered--Jülich station federal funding proposal submitted
• LOFAR-UK:– Chilbolton, Cambridge, Jodrell, Edinburgh– 1 station funded, site to be chosen--3 more stations
• Sweden:– Onsala station funded
• + Poland, Italy, Austria, Bulgaria, Ireland, France
EURO-VO: GRID and VO
Wires in the fieldWires in the field
EURO-VO: GRID and VO
EURO-VO: GRID and VO
Station based processing
Station based processing
–Receiver–Filter–Beamformer
– Bu
ffer
–Receiver–Filter–Beamformer
– Bu
ffer
–Receiver–Filter–Beamformer
– Bu
ffer
–Receiver–Filter–Beamformer
– Bu
ffer
–10 GbE
–1 GbE
–Used for 1 beam & station cross correlation
Input data rate: ~ 460 Gbps
Output data rate: ~ 2 Gbps
Processing capacity: ~ 1.5 Tmul/s
Storage capacity: 96 Gbyte
Lofar: Data StreamLofar: Data Stream
Gb
E s
witc
h
Gb
Esw
itch
Gb
Es
witc
hG
bEsw
itch
Gb
Esw
itch
Gb
Esw
itch
Auxiliary processingsection
Sto
rag
e sy
ste
m
Off-line processing/user software
Off-lineStorageOn-Line
Long term Archiving& Data Export
Gb
Esw
itch
On-Line pipeline processingBlue Gene/L
WAN
EURO-VO: GRID and VO
EURO-VO: GRID and VO
Lofar StorageLofar Storage
Lofar Long term (>2weeks)archive and user enabling
ADD
Lofar Long term (>2weeks)archive and user enabling
ADD
EURO-VO: GRID and VO
STREAM of MetaDataSTREAM of MetaData
• Design: data driven vs. compute driven• from the field to central processor• Distributed – p2p / federated
– Communities– Dataservers – own + EGEE GRID– Compute servers – own+ EGEE GRID
• AstroWISE• EURO - VO
• Design: data driven vs. compute driven• from the field to central processor• Distributed – p2p / federated
– Communities– Dataservers – own + EGEE GRID– Compute servers – own+ EGEE GRID
• AstroWISE• EURO - VO
EURO-VO: GRID and VO
EURO-VO: GRID and VO
LoWISE designLoWISE design
Lofar-WISE: Data StorageLofar-WISE: Data Storage
–Groningen
–SARA AmsterdamEURO-VO: GRID and VO
• IBM – GPFS filesystem• Connect to AstroWISE
– Storage table
• IBM – GPFS filesystem• Connect to AstroWISE
– Storage table
EURO-VO: GRID and VO
LoWise- conceptsLoWise- concepts• All Tier 1s on an equal footing• Centrally operated GPFS distributed filesystem• File retrieval
– Db viewer (web)– Prompt (db query)– Pathname – data server– gLite – Grid ftp
• Single distributed db- with all Meta and source data– Valid --Quality -- ready for publication flags
• Visibility of data– Owner –Project –Archive --EURO-VO
• Authentication / Assigned resources– AstroWISE Oracle system
• All Tier 1s on an equal footing• Centrally operated GPFS distributed filesystem• File retrieval
– Db viewer (web)– Prompt (db query)– Pathname – data server– gLite – Grid ftp
• Single distributed db- with all Meta and source data– Valid --Quality -- ready for publication flags
• Visibility of data– Owner –Project –Archive --EURO-VO
• Authentication / Assigned resources– AstroWISE Oracle system
EURO-VO: GRID and VO
Processing - optionsProcessing - options
• Tier 0 – CP• Tier 1 – re-processing
– Local clusters– EGEE GRID
• Tier 2– On Tier 1– Local anarchy
• Tier 0 – CP• Tier 1 – re-processing
– Local clusters– EGEE GRID
• Tier 2– On Tier 1– Local anarchy
EURO-VO: GRID and VO
EURO-VO: GRID and VO
AstroWISE-UpAstroWISE-Up
EURO-VO: GRID and VO
–VST - Virtual Survey Telescope–VST - Virtual Survey Telescope
EURO-VO: GRID and VO
– www.astro-wise.org– www.astro-wise.org
EURO-VO: GRID and VO
Astro-WISE: ServicesAstro-WISE: Services
EURO-VO: GRID and VO
TARGET diagram TARGET diagram
EURO-VO: GRID and VO
EURO-VO: GRID and VO
QC - calibration scientist monitoring
QC - calibration scientist monitoring
EURO-VO: GRID and VO
Extreme data lineage Extreme data lineage
Astro-WISE: Grid Computing
Astro-WISE: Grid Computing
–selection of Target in Astro-Wise
–storage element defined a priori –EGEE computing EURO-VO: GRID and VO
EURO-VO: GRID and VO
LoWise and Euro-VOLoWise and Euro-VO
EURO-VO: GRID and VO
EURO-VO: GRID and VO
GRID- EuroVO –Awe visualization
GRID- EuroVO –Awe visualization
EURO-VO: GRID and VO
– AI – Cultural Heritage – AI – Cultural Heritage
EURO-VO: GRID and VO
http://omegate.astro.rug.nl:9001/Monk
http://omegate.astro.rug.nl:9001/Monk