Date post: | 04-Jan-2016 |
Category: |
Documents |
Upload: | rudolf-davidson |
View: | 214 times |
Download: | 0 times |
EURO-VO workshop 1 July 2005
A Virtual Survey SYSTEMA Virtual Survey SYSTEM
Astro-Wise
NOVA/Kapteyn – OA CapodimonteESO – Terapix – US Munich/MPE
National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO
5 year programme -> dec 2006
Astro-Wise
NOVA/Kapteyn – OA CapodimonteESO – Terapix – US Munich/MPE
National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO
5 year programme -> dec 2006
Edwin A. ValentijnEdwin A. Valentijn
EURO-VO workshop 1 July 2005
Basic objectivesWide Field imaging EU
Basic objectivesWide Field imaging EU
Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results
100’s Tbyte of image data and 10’s Tbyte of catalogue dataWith production spread over EU
What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)
Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results
100’s Tbyte of image data and 10’s Tbyte of catalogue dataWith production spread over EU
What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)
(O)MegaCAM(O)MegaCAM
EURO-VO workshop 1 July 2005
statusstatus
• VO conference 2002 - design• Build information system - working• Implemented, qualified [email protected], WFC@INT, MDM,
OmegaCAM@ILT• >>Qualify with OmegaCAM@VST-2006• >>tune to run Public Surveys• >>quality control• >>Optimize federation/replication
• VO conference 2002 - design• Build information system - working• Implemented, qualified [email protected], WFC@INT, MDM,
OmegaCAM@ILT• >>Qualify with OmegaCAM@VST-2006• >>tune to run Public Surveys• >>quality control• >>Optimize federation/replication
EURO-VO workshop 1 July 2005
new paradigmtarget processing
new paradigmtarget processing
from:-waterfall/ multi-tier -data pushing-raw data processing-raw data delete-result ->archive -”releases”
from:-waterfall/ multi-tier -data pushing-raw data processing-raw data delete-result ->archive -”releases”
to:-hunting /full linking-data pulling-target processing-raw data -archive -all in archive -request driven
to:-hunting /full linking-data pulling-target processing-raw data -archive -all in archive -request driven
raw pixel data pipelines/cal files cataloguesall integrated in one information system
• distributed services Virtual Survey Telescope• processing GRID • Storage GRID • Methods/services GRID
raw pixel data pipelines/cal files cataloguesall integrated in one information system
• distributed services Virtual Survey Telescope• processing GRID • Storage GRID • Methods/services GRID
EURO-VO workshop 1 July 2005
Astro-Wise VO Properties Benefits integrated dynamic db
Astro-Wise VO Properties Benefits integrated dynamic db
• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO
• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO
EURO-VO workshop 1 July 2005
componentscomponents
• Procedures + Cal plan at telescope• Data model -> object model ++ ->++db• Central db ; server/clients
– All I/O except images– Meta data– Source lists = catalogues + associate lists– Links = references = joints
• Fileserver – distributed- via db• Python clients• CVS distributed code base - opipe
• Procedures + Cal plan at telescope• Data model -> object model ++ ->++db• Central db ; server/clients
– All I/O except images– Meta data– Source lists = catalogues + associate lists– Links = references = joints
• Fileserver – distributed- via db• Python clients• CVS distributed code base - opipe
EURO-VO workshop 1 July 2005
Astro-Wise PipelinesAstro-Wise Pipelines
Photometric pipelinePhotometric pipeline
Bias pipeline
Flatfield pipeline
Image pipeline
Source pipeline
EURO-VO workshop 1 July 2005
Target processing:++ the make metaphor
Target processing:++ the make metaphor
awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382')
The processing chain is
ReadNoise <-- Bias <-- HotPixels
> class HotPixelMap(ProcesTarget): > > def self.make()
> class ProcessTarget(): > > def get(date, chip) # if not exist/up-to-date then make() > > def exist() # does the target exist? > > def uptodate() # is each dependency up to date?
Fully recursive
awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382')
The processing chain is
ReadNoise <-- Bias <-- HotPixels
> class HotPixelMap(ProcesTarget): > > def self.make()
> class ProcessTarget(): > > def get(date, chip) # if not exist/up-to-date then make() > > def exist() # does the target exist? > > def uptodate() # is each dependency up to date?
Fully recursive
EURO-VO workshop 1 July 2005
Intra-operability peer to peerIntra-operability peer to peer
• code base + docs : CVS
• Db: “Advanced Replication” evolving to streaming
• code base + docs : CVS
• Db: “Advanced Replication” evolving to streaming
WRITE
–READ-ONLY
–READ-ONLY
–RE
AD
-ON
LY
–RE
AD
-ON
LY
WRITE
–REPLICATIO
N
–REPLICATION
–RE
PLIC
AT
ION
–RE
PLI
CA
TIO
N
EURO-VO workshop 1 July 2005
Contents of federationContents of federation
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Co added images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Co added images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
EURO-VO workshop 1 July 2005
Example 5LSExample 5LS
#Find ScienceFrames for a ccd named ccd53 and filter
Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’)
# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]
# get the rms of the used MasterflatAwe> y = [k.flat.imstat.stdev for k in q]
# Make a plot Awe> pylab.scatter(x,y)
#Find ScienceFrames for a ccd named ccd53 and filter
Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’)
# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]
# get the rms of the used MasterflatAwe> y = [k.flat.imstat.stdev for k in q]
# Make a plot Awe> pylab.scatter(x,y)
EURO-VO workshop 1 July 2005
QC - calibration scientist monitoring
QC - calibration scientist monitoring
EURO-VO workshop 1 July 2005
QC - calibration scientist monitoring
QC - calibration scientist monitoring