MODEG, Brussels 2009-11-25

5000 Years of Libraries 50 Years of Data Centers

PANGAEA®

Michael Diepenbroek, Hannes Grobe, Uwe Schindler

MODEG, Brussels 2009-11-25

What is PANGAEA® ?

• since 1993Information system for earth system science data hosted by AWI & MARUM (estimated yearly budget ~ 1 Mio €)

• 2001 Mandate of the International Council for Science (ICSU): World Data Center for Marine Environmental Sciences (WDC-MARE)

• 2007Mandate of the World Meteorological Organisation (WMO): World Radiation Monitoring Center (WRMC)

MODEG, Brussels 2009-11-25

PANGAEA® - activities & services

• Project data management

• Long term data archive (data library)

• Data publication

• Data infrastructures

MODEG, Brussels 2009-11-25

Project data management

CARBOOCEAN CORALFISH EPOCA EUR-OCEANS ESONET / EMSO (ESFRI) HERMES / HERMIONE HYPOX IODP SPICOSA JGOFS international TARA OCEANS…

More than 90 European to international projects since 1995(www.pangaea.de/projects)

MODEG, Brussels 2009-11-25

Long term archive

• Open access & non restricted data Creative Commons license

• Data accepted from individual scientists, institutes, and science projects

• Long term funding for basic operation hardware, software, system management & organisation

• Long term preservation of data Technical: security, migration of media, Usability: preserving the integrity & semantics of data sets

MODEG, Brussels 2009-11-25

Long term archive - contents

IRD

( gr av/ 10 cm 3)

Sand

( %)

CaCO3

( %)

TOC

( %)

Radio

( %/ sand)

Smect

( %/ clay)

IRD

( gr av/ 10 cm 3)

Sand

( %)

CaCO3

( %)

TOC

( %)

Radio

( %/ sand)

Smect

( %/ clay)

IRD

( gr av/ 10 cm 3)

Sand

( %)

CaCO3

( %)

TOC

( %)

Radio

( %/ sand)

Smect

( %/ clay)

IRD

( gr av/ 10 cm 3)

Sand

( %)

CaCO3

( %)

TOC

( %)

Radio

( %/ sand)

Smect

( %/ clay)

IRD

( gr av/ 10 cm 3)

Sand

( %)

CaCO3

( %)

TOC

( %)

Radio

( %/ sand)

Smect

( %/ clay)

PS1389-3 PS1390-3 PS1431-1 PS1640-1 PS1648-1

Age (kyr) max. : 233.55 kyr PS1389-3ff

0.0

100.0

200.0

0 20 0 100 0 15 0 0. 5 0 50 0 100 0 20 0 100 0 15 0 0. 5 0 50 0 100 0 20 0 100 0 15 0 0. 5 0 50 0 100 0 20 0 100 0 15 0 0. 5 0 50 0 100 0 20 0 100 0 15 0 0. 5 0 50 0 100

54° 0' 54° 0'

54°30' 54°30'

55° 0' 55° 0'

55°30' 55°30'

11°

11°

12°

12°

13°

13°

14°

14°

15°

15°

World vector shore lineGrain size class KOLP AGrain size class KOEHN2Grain size class KOEHNGeochemistryGrain size class KOLP BGrain size class KOLP DIN20 m

Scale: 1:2695194 at Latitude 0°

Source: Baltic Sea Research Institute, Warnemünde.

• Profiles -> doi:10.1594/pangaea.103958

• Time series -> doi:10.1594/pangaea.323487

• Sea bed photos -> doi:10.1594/PANGAEA.319877

• Distributes samples -> doi:10.1594/pangaea.51749

• Complex data -> doi:10.1594/PANGAEA.108079

• Air photos -> doi:10.1594/PANGAEA.323540

• Audio record -> doi:10.1594/PANGAEA.339110

MODEG, Brussels 2009-11-25

unclassified

Atmosphere

Corals

Ice

Sediment

Water

Total number of data sets ~ 585,000 Data items ~ 5.1 billions

Long term archive – contents (11/2009)

MODEG, Brussels 2009-11-25

Conform to global standards (GSDI) ISO19xxx, OGC, W3C, OAI

Portals CARBOOCEAN (4) EUR-OCEANS (16) IODP - SEDIS (4) World Data Center portal (15)

Broker function GBIF, OBIS

Data warehouse

Data infrastructures

MODEG, Brussels 2009-11-25

PANGAEA® – standard interfaces for metadata

data management & longterm archiving

RDB

catalogues

PANGAEA

ISO19xxx

STD-DOI

XSLT

Index

Dublin Core

protocols

marshaller

WS(SOAP/WSDL)

Frontends / portals

PANGAEAweb frontend

Geoserver(OGC)

OGCcatalogue

service

OAI-PMH

ISO690

GeoPortal.Bund®

TIB Library

WS(SOAP/WSDL)

DOI registration

catalogues

DOI registry

DIF DublinCoreharvester

Google

OCLC

harvester

GCMD

EUR-OCEANS

CARBOOCEAN

IODP

Darwin Core

DiGIR Darwin Core

ISO19xxx

DIF

OBIS

GBIF

harvester

harvester

D-GRID

gml, kml

WDS

MODEG, Brussels 2009-11-25

PANGAEA®– Verbreitung von Daten und Metadaten

MODEG, Brussels 2009-11-25

Data publication

• following the OECD principles and guidelines for access to research data (2007)

• peer-reviewed citable data sets referenced by persistent identifiers (DOI)DOI registry -> crossref for scientific data

• Collaborations with publisherswith data journalscrossreferencing supplementary data with traditional publications

(SCOR working group, Elsevier, Nature, Springer, Thompson Reuters)

MODEG, Brussels 2009-11-25

QA/QCby data center

peer review(incl. data)

send data sets

archive data sets

send identifier(accession number)

publish data sets

send article

publish article

prepare article &related data sets

DATA CENTRE

author,data originator

editor

reviewersdata curator

yes

no

accepted?

yes

accepted?

JOURNAL

no

Supplementary data

Partnership with Elsevier

MODEG, Brussels 2009-11-25

MODEG, Brussels 2009-11-25

MODEG, Brussels 2009-11-25

•Nuclear RadiationTokyo, Japan

WDC Co-ordination OfficesWashington DC, USABeijing, China

•MeteorologyAsheville NC, USABeijing, ChinaObninsk, Russia

•OceaographyObninsk, RussiaSilver Spring MD, USATianjin, China

•PaleoclimatologyBoulder CO, USA

•Marine Geology and GeophysicsBoulder CO, USAMoscow, Russia

•Remotely Sensed Land DataSioux Falls SD, USA

•Renewable Resources and EnvironmentBeijing, China

•Recent Crustal MovementsOndrejov, Czech Republic

•AirglowMitaka,Japan

•AstronomyBeijing, China

•Atmospheric Trace GasesOak Ridge TN, USA

•AuroraTokyo, Japan

•Cosmic RaysToyokawa, Japan

•GeologyBeijing, China

•Human Interactions in the EnvironmentPalisades NY, USA

•IonosphereTokyo, Japan

•Earth TidesBrussels, Belgium

•GeomagnetismCopenhagen, DenmarkEdinburgh, UKKyoto, JapanColaba, India

•GlaciologyBoulder CO, USACambridge, UKLanzhou, China

•Marine Environmental SciencesGermany, (2001)

•Rotation of the EarthObninsk, RussiaWashington DC, USA

•Satellite InformationGreenbelt MD, USA

•Rockets and SatellitesObninsk, Russia

•SeismologyDenver CO, USABeijing, China

•Solar Radio EmissionNagano, Japan

•Space ScienceBeijing, China

•Space Science SatellitesKanagawa, Japan

•Solar ActivityMeudon, France

•SoilsWageningen, The Netherlands

•Sunspot IndexBrussels, Belgium

•Solar Terrestrial PhysicsBoulder CO, USADidcot Oxon, UKMoscow, RussiaHaymarket, Australia

•Solid Earth GeophysicsBeijing, ChinaBoulder CO, USAMoscow, Russia

ICSU World Data Centers (WDC)Geophysical Year 1957

MODEG, Brussels 2009-11-25

GEOSS Global Earth Observation System of Systems

The missing link !?

MODEG, Brussels 2009-11-25

Contra• Overall data availabity is low compared to data production

• Organisation and quality of data is lacking consistency

• IT development is fast – no time for legacies

• Fragmentation of efforts

Pro• Long standing experience & know how & motivation

• Good context with science

• Open access for all data resources

• As a whole a very large global data management capacity

• Trans-disciplinary !

Initial position of ICSU WDS

MODEG, Brussels 2009-11-25

ICSU WDS - Roles & relations in a federated system

Publishers commercial, open access

(e.g. ESSD journal), crossreferencing

Data Collection & Processing FacilitiesQA/QC, data products, also

data rescue

Data Archiving & Publication Facilities

certified repositories

Related Networks & Programs

GEOSS, GMES, WMO-IS, IOC etc

Metadata & Data Services

web portals, catalogues

Visualisation & Analysis

compute systems, virtual labs, GIS systems

Research Institutionsuniversities,

research institutes

Research Projects / Programsnational, EU, international

Libraries DOI registry

interdiscipl. catalogues

Research Facilitiessattelites, vessels,

observatories, alert systems etc.

Education & Outreach

MODEG, Brussels 2009-11-25

ICSU WDS - further steps

• Survey collecting responses from interested parties Preliminary results indicate that the consortium will partly change

• Evaluation of old and new facilities• Active recruitment of new candidates• Certification procedure

Catalogue of criteria, workflows, structures (OAIS) Certification authority (CA)

• Relationships with GEOSS, WMO-IS

MODEG, Brussels 2009-11-25

Overall concept & context with EMODNET

Data & Compute CentersNODCs + WDCs + specialized Thematic Data and Service Centers

Data providers & ConsumersESONET, CARBOOCEAN, HERMES, EUROSITES, EuroGOOS, SESAME, EPOCA, MEECE, CoralFish, BASIN etc.

Management & Advisory Network based on European virtual institutes (EMSO, MarBef, EUR-OCEANS, Marine Genomics, SPICOSA) + relevant international programs & bodies (IMBER, CLIVAR, SCOR etc.)

EMODNet