ESA Scientific Archives andVirtual Observatory Systems

Christophe.Arviset@esa.intScience Archives and VO Team

Research and Scientific Support DepartmentESA – ESAC – Madrid, Spain

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 2 Research & Scientific Support Department

European Space Astronomy Centre

ESAC default location for: Science operations,

long history with astronomical missions,

now expanding with solar system missions.

Science archives, Astronomy Planetary

ESA VO activities, ESAC to be the European VO

node for space-based astronomy.

http://www.esa.int/SPECIALS/ESAC/Located near Madrid, Spain

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 3 Research & Scientific Support Department

Archives at ESAC ESAC is the Centre where most of ESA Scientific Directorate’s Scientific

Archives are developed, maintained and operated.

ESAC Science Archives Team is giving support to various projects

ISO Data ArchiveSince December 1998

XMM-Newton Science Archive

Since April 2002

Integral SOC Science Data Archive

Since July 2005

Planetary Science ArchiveGiotto, Mars Express

Rosetta, Venus ExpressSmart-1, Huygens

Since March 2004

Herschel, Planck, GAIA, … in the future

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 4 Research & Scientific Support Department

Archives at ESAC (cont’d) Scientific Archives geared to various users

Scientific Community (public access) PI team and observers (controlled access) Science Operations Team (privilege access)

Common Architecture and Look and Feel Better corporate image for ESA Faster, cheaper, better development Re-use of expertise

User friendly access Eg PSA Mars Map Browser

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 5 Research & Scientific Support Department

Science Archives Team @ ESAC A core Science Archives and VO Team (~15 people) in ESAC

Support many projects (horizontal support vs vertical organization)• ISO, XMM-Newton, Integral, Planetary missions, Herschel, • Virtual Observatory• Soho, Planck, Gaia in the future…

Organized by functions: database, user interface, data distribution, inter-operability

Close collaboration with Archive Scientist (1 per project) Expert of the science field, close to the community Consolidate user requirements Set development priorities Final acceptance tests

Strong re-use of design/system/code between projects

Not responsible for writing data reduction software (pipeline) nor data production

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 6 Research & Scientific Support Department

Code re-use across Archives

Manpower shared between projects Knowledge transfer between projects Small fraction of FTE for “old” projects

All Archives share a common part of the code Same functionalities across projects But still some very specific projects functionalities Faster and reliable development of new projects Easier and cheaper maintenance of existing systems

Same Look & Feel for ESA Science Archives User Interface Powerful and easy to use Better corporate image for ESA Easy learning curve for users

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 7 Research & Scientific Support Department

Data Repository

All Data from all archives are stored on magnetic disks Standard formats (FITS for astronomy, PDS for planetary) Save various levels of data IDA (0.5TB), XSA (3TB), ISDA (6TB), PSA (4TB) Small amount still

Unique disk repository for proprietary and public data Access controlled by the application server

Access to data is immediate Via the Archive User I/F or for the VO Except for on-the-fly reprocessing requests

Clever FTP server using links instead of real files Same disk space for FTP Higher quotas for users

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 8 Research & Scientific Support Department

DataBase

Strong effort of Data Modeling before starting any project

Unique database for all uplink and downlink metadata

All metadata is systematically extracted from uplink AND downlink data Done when products are generated Can be re-run with for new metadata is user reqs change

Sybase for some projects ISO (10GB), XMM-Newton (6GB), Planetary (2GB)

Oracle for others Integral (6GB)

PostGres for future projects (Herschel, Soho, …)

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 9 Research & Scientific Support Department

Existing Access – GUI

Project

Business

Logic

TranslationLayer

Meta Data in

Database

Data Productson hard disks

ProductRequest

Products

GUI Database/Product Request

GUI Database Result / Product

DB/Product Request

DB Result / Product

User friendly web access : Java Applet Powerful queries, organized by panel,

hundreds of queriable and displayable parameters

Public data accessible to all, proprietary data accessible only to privilege users

Images preview (icons, full image) Quick download (1 click) at various processing

levels, shopping basket retrieval Remote access to other archives (ADS, and

tools (Aladin, VOSpec)

Project prior to the VO Existing data model, data format, access

mechanisms

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 10 Research & Scientific Support Department

Existing Access : Scriptable I/F

Project

Business

Logic

TranslationLayer

Meta Data in

Database

Data Productson hard disks

ProductRequest

Products

GUI Database/Product Request

GUI Database Result / Product

DB/Product Request

DB Result / Product

Scriptable

Interface

Project Product Request

Project Productforeach obs in (observation_list)        AIOgetfile (obs, file_required)        analyse_process file_required        save results        delete file_requiredend

DB/Product Request

DB Result / Product

Access to metadata (via XML scripting) Access to data through socket (direct data stream) or

FTP (several files or individual file)

Own Interoperability standards and needs as required by the project

Not using VO protocols, but ready for it…

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 11 Research & Scientific Support Department

VO compliant Architecture

Project

Business

Logic

TranslationLayer

Meta Data in

Database

Data Productson hard disks

ProductRequest

Products

GUI Database/Product Request

GUI Database Result / Product

DB/Product Request

DB Result / Product

Project Product Request

Project Product

DB/Product Request

DB Result / Product

SIAP/SSA Request

VOTable / Product

TranslationLayer

Scriptable

Interface

Project Product Request

Project Product

(Scriptable Interface)foreach obs in (observation_list)        AIOgetfile (obs, file_required)        analyse_process file_required        save results        delete file_requiredend

DB/Product Request

DB Result / Product

Existing scriptable I/F already carried all the functionalities required by the new VO standards Access to metadata (database) Access to products (data repository)

By using “Translation Layer”, easy to convert the VO standards interfaces into our existing I/F Little effort required Translation Layer easy to code using XML

As VO standards evolve, our scriptable I/F system remains, but we just have to adapt our translation layers

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 12 Research & Scientific Support Department

ESA-VO Main Goals ESA part of the EURO-VO ESA part of the IVOA (International VO Alliance)

RSSD astronomical archives being VO compatible VO in mind when building archives Archives and VO work done by the same team

Develop VO applications eg VOSpec, VOQuest, …

Develop VO publishing services Registry of VO resources, DALToolkit, DMMapper, …

ESAC to disseminate VO technology Link to existing and new ESA astronomy

projects VO outreach (workshop, conferences, …)

http://esavo.esac.esa.int/

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 13 Research & Scientific Support Department

VO in action : VOSpec application

Select Source

Find/Select in the VORegistry

VO Spectral Servers

IUEspectra

ISOspectra

VEGATheoretical

Model

DisplayTheoretical Spectra

and Observational Spectra

IUEspectra

ISOspectra

VEGATheoretical

Model

NormalizeTheoretical

Spectra

IUEspectra

ISOspectra

VEGATheoretical

Model

Theoretical SpectraFits

Observational Spectra

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 14 Research & Scientific Support Department

Interoperability ESA PSA – NASA PDS

(Prototype in development) in the context of the IPDA (International Planetary Data Alliance)

From Mars Map Browser, Select region of interest

Contact PSA and PDS using the common data (image) access protocol

Display NASA PDS and ESA PSA images

Christophe ARVISETScience Archives in the 21st Century, 26/04/2007, page 15 Research & Scientific Support Department

Conclusion

ESA Scientific Archives built and operated within a same team

ESA Archives flexible architecture allows easier and long-term maintenance

ESA strongly believes in the Virtual Observatory, both for astronomy and for planetary missions

Strong link between Archives development and Virtual Observatory work

ESAC is becoming the Data Centre for ESA space-based archives

ESAC is becoming the VO node for European space-based astronomy