Post on 21-May-2020
transcript
Seismology and EPOS Integrated Core Services
Daniele Bailo (INGV) and the EPOS IT team
ORFEUS meeting – September 22, 2015 - Bucarest
What is EPOS? EPOS is a long-term plan for the integration
of research infrastructures for solid Earth Science in Europe
EPOS integrates the existing (and future) advanced European facilities into a single, distributed,sustainable infrastructuretaking full advantage of new e-science opportunities
EPOS Architecture
Governance
Financial Work
Technical Implementation
Communication
Harm
onization
EPOS IP project Timeline
Implementation Validation Pre-operation
Communities and services
Communities and services
Communities in numbersMAP OF:- Seismic/GPS stations- Laboratories-- etc….
Diversity in data type and formats
http://www.epos-eu.org/ride/
Research InfrastructureLIst
• 244 Research Infrastructures
• 138 Institutions• 22 countries• 2272 GPS receivers• 4939 seismic stations• 464 TB Seismic data• 1.095 PB Storage
capacity (seismology)• 1.240 PB Storage
capacity (GNSS DC)• 828 instruments in 118
Laboratories
The challenge of integration
Storage / curation challenge
Yesterday Today
Data exchange challenge
Yesterday Today
Format / standard challenge
Yesterday Today
quakeML
SEED
ASCII filesBinary filesLocal metadataNo metadata
TelegraficFormat
IASPEI Seismic Format (ISF)SAC
SEED
stationML
Dost, B., Zednik, J.,et al. (2012): Seismic Data Formats, Archival and Exchange. - In: Bormann, P. (Ed.), New Manual of Seismological Observatory Practice 2 (NMSOP-2), Potsdam : Deutsches GeoForschungsZentrum GFZ, p. 1-19.
Future challenges: do you know …• Who (name and surname) and how is
using your data?• Can you measure the impact on society?• Which is the “value” of your data?• Can I use it for commercial purposes?• Can I make computation without
downloading waveforms?• Will the data / software survive you?
Future challenges: do you know …• How to increase the usage of your data?• How to increase the citation of your
data/DC/publications?• How to improve system performance?• How to bridge with other disciplines (e.g.
seismic data from volcanologist)?
…the answer is …….
INTEGRATION
Advantages of integration• USERS: access to an environment where
heterogeneous data and tools available:– Multidisciplinary science made simple [1]– More papers, more citations– Data and tool availability
• DATA PROVIDERS:– Great diffusion of data (EU funds attraction?)– Track usage of data and services– steering capacity– EU open data policy– do not reinvent the wheel
[1]: [Chiaraluce, L. (2012). Unravelling the complexity of Apenninic extensional fault systems : A review of the 2009 L ’ Aquila earthquake Journal of Structural Geology. ]
HOW TO achieve (technical) INTEGRATION for solid Earth
data/services?A) By means of discussions, conferences,
meetingsB) Promoting interoperability of systemsC) By setting up a new international
standardization bodyD) Promoting participation to European projects
Interoperability“Promoting interoperability of systems”
… What does it mean?
1.Connect several systems (bricks)2.By means of interfaces (APIs or
web services)3.No need to know system interal
specifics (black box concept)
Interoperability: example
FDSN station FDSN dataselect FDSN events GSAC services
Seismic Waveforms Seism Events GPS wavef.
THIN LAYER (or mediator, or EPOS integrated services or…)
Interoperability implies usage of standards:• For metadata• For data formats• For communication protocols
FuturePlatform
Scientist!
Accelerometric Data
SAR Data
Geodetic DAta
ENVISAT_987987998_add | show | info
Lipari Islands M:2.22013.08.23 - add | show | info
Sicily North East Coast M:2.12013.08.21 - add | show | info
Sicily North East Coast M:2.32013.08.18 - add | show | info
Rynex ad098098 2009.04.09 - add | show | info
Sicily North East Coast
square
squareENVISAT_43287987998_6add | show | info
ENVISAT_987987998_Switch to advanced search
Rynex ad098098 2009.04.09 - add | show | info
Rynex ad098098 2009.04.09 - add | show | info
Latest Earthquakes
Shakemaps
Hystorical Earthquakes
Documents & other resources
Sicily North East Coast2013.08.15 - Download
Sicily sea2013.05.14 - Download
Civil protection: Sicily hazards2013.02.01 - Link
Hystory of Sicily Hearthquakes2013.02.01 - Link
Lipari Islands M:2.22013.08.23 - Info
Sicily North East Coast M:2.12013.08.21 - Info
Sicily North East Coast M:2.32013.08.18 - Info
Siracusa Aea - Imax: 11MCS1693.01.11 - Infos
Sicily North East Coast
Accelerometric DataMI05 07-04-09 17:47 5.5AQV 06-04-09 01:32 6.3AQG 06-04-09 01:32 6.3AQA 06-04-09 01:32 6.3AQK 06-04-09 01:32 6.3PCB 09-04-09 19:38 5.2AQV 07-04-09 21:34 4.5AQM 06-04-09 02:37 5.1AQU 06-04-09 01:32 6.3PCB 09-04-09 00:53 5.4GSA 07-04-09 17:47 5.5PCB 13-04-09 21:14 5.0
SAR Data
Remove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | Show
Bulk Actions
Accelerometric WaveformsFilter
Hi-PassLow-PassBand-Pass
ENVISAT-1-09083009242556-1600.ASA_IM__0PENVISAT-1-09072609242628-1600.ASA_IM__0PENVISAT-1-09062109242691-1600.ASA_IM__0P
Accelerometric Data:
SAR Data:
• Integrate use of SAR, GPS, Accelerometric Data, etc.• Use different codes and languages (python, fortran, any other…)• Perform heavy processing online (use of HPC resources)• Compare results (e.g. focal mechanism catalogues) • Compare different data• Save data in personal area•…and more… (even download the data)
Scientist at work
Then… what can the scientist do?
Generic Features1. Download
2. Visualization
3. Processing
4. Modeling
GPS DATADOWNLOAD
SEISMIC DATA DOWNLOAD
EPOS demonstrator
Collaboration with VERCE http://epos.cineca.it
IT Architecture & Design
Architecture
Thematic Core Services(TCS)
Integrated Core Services(ICS)
Interoperability layer
TCS generic architecture
TCS system Data/metadata catalogue
National network
Nationalrepository
API / web service
Local HPC
EPOS ICS
PrinciplesCo-development (ICS and TCS)
• The development of the ICS depends on the requirements and provided data, services, resources and users of the TCS. The TCS are at different stages of maturity. For some – with little infrastructure to date - the adoption of the EPOS architecture is straightforward. For others – with several years of infrastructural investments behind - a jointly agreed evolutionary plan to converge to interoperability with ICS will be adopted. The EPOS approach in this context is neither top-down nor bottom up: the main idea is to use the general architecture and follow a cooperative approach in the designing and development of the software to build the compatibility layer, which is the place where harmonisations and communications are achieved.
PrinciplesDo not reinvent the wheel
• a. Reuse local technologiesThe co-development philosophy maximises re-use of existing software services, data availability and resources. (e.g. VERCE, EUDAT)
• b. Do not build a supercomputer: build ICS-DEPOS will have its own computing resources mainly to provide the ‘uniform view’ over EPOS entities. For the massive processing EPOS will provide – subject to authorisation – access to appropriate computing facilities including HPC (High Performance Computing) machines for modelling and simulation, and HTC (High Throughput Computing) clusters for data intensive applications such as data mining. These facilities are known collectively as ICS-D (Integrated Cores Services – Distributed)
• c. EU Initiatives Harmonization: EGI, ENVRI, EUDAT, RDA…
Principles
Microservices approach• The Microservices architecture approach envisages small
atomic services dedicated to the execution of a specific class of tasks, which have high reliability. EPOS will either take existing software services and ‘wrap’ them for EPOS use or build new services complying with EPOS architectural standards. The aim is to have atomic services with defined interfaces which can be composed together to form a software stack able to address unpredictable user request.
Principles
Clear long-term technical goals, but iterative short-term approach
• The overall architecture of EPOS is clearly defined and agreed through EPOS-PP. However, its implementation in EPOS-IP would require a step-by-step approach to build a reliable system environment to meet the requirements of end-users and their communities
Principles – lessons learnt
• Working in phase (ICS and TCS)• Communicating across different domain (IT,
scientists, lawyers ..etc)• Collaborating and building communities
Thank You
www.epos-eu.org
WebSite
www.epos-eu.org/ride
R.I.D.E.
Newsletter
www.epos-eu.org/newsletter
Epos Social
daniele.bailo@ingv.it
IT Architecture & Design
Architecture
Thematic Core Services(TCS)
Integrated Core Services(ICS)
Interoperability layer
ICS - Integrated Core ServicesICS-C and ICS-D
ICS-C ICS-D• ICS-Central Hub• Orchestrates external
resources (ICS-D)• EPOS member states
competition for hosting it
• ICS-Distributed services• Provide IT resources not
built by EPOS ICS-C• Computational resources• Visualization resources• Others…
EPOS GUI
METADATAcatalogue
EPOS APIs
System manager
Dataset
Services
Software
TCS API
AAI system
Workflow Engine
ICS-D
ICS-C
TCS
NationalLayer
IT Seismic net FR Seismic
net
Dataset
Services
Software
TCS API
PT GPS net NO GPS
net
…Other TCSs
Elements of EPOS ecosystem
Complete ICT environment for research
Researchers, research managers, innovators, media
Processing Model
User Model
Data Model
Resource Model
User interaction with the system:AAAI, interaction workflow, multlinguality
Describes functions of processing environment etc.
Describes research dataand ancillary info (instituions)
Representing ICTresources
Metadata model- 3-layers- Manage the complexity
(see previous slide)- CERIF (formal syntax,
declared semantics)Functions-Discovery, selection-Impact assessment-Manage interoperabilityIssues (selection of)- heterogeneity- Many standards to be
mapped from TCS- Ontologies- Complex work
Discovery (DC) and (CKAN, eGMS)
Contextual (CERIF metadata model)
Detailed (community specific)
Web portal, Spatio-Temporal Search
domain specific – data(TCS or Naational Ris)
Contextual (CERIF metadata model)
(http://www.eurocris.org/)Common European Research Information Format
EPOS Interface
Human interface (GUI)Website or portal
Machine interface- API or Web service - which execute scripts
or queries- Returns data/metadata
in a given standard
EPOS APIs
CERIF XML APIs
Metadata avaialble in different formats:• RDF export (“old” ENVRI)• OAI-PMH, CKAN, opensearch… (EUDAT)• other standards
OAI-PMHRDF …
EPOS Interface: interoperability
Ontologies
Machine readable andMachine understandable
How does it work?1. Connects concepts2. Needs vocabulary
Issues• Many ontologies exist• Vocabulary Mapping • CERIF supports it
Michelini
CNT
Is Director of
INGV
Is section of Gresta
Is president of
Sailing
Has hobby
Trieste
Is Born
Italy
Located in
Boat
use
sea
use
AAI system (federeated & distributed)
Purpose - Authenticate users- Authorize users - Transparent access to
TCS and ICS-D data & services
Issues- Delegation- Many system,
sometimes non interoperable
- EPOS-EGI-CC- AARC
- X509- EPOS IdP- TCS need guidelines- Accept all auth.
mechanisms
Workflow engines
Purpose - Tracks data
transformation- Allows versioning- Allows reproducibility
Comments- Interoperability among
various workflow engines- VERCE contribution- E-FLOW by EGI- CERIF ??
How to register/cite data or publications?PID system
Purpose - DO / publication can be
uniquely referenced- Assign a PID at data
creation times
Issues- Need for a simple
mechanism to implement it
- EUDAT can help- Not at ICS level (only for
TCS?)
TCS envisaged architecture
TCS system Data/metadata catalogue
National network
Nationalrepository
API / web service
Local HPC
EPOS ICS
ICS-TCS common work
Metadata• Definition of community, standard-
based metadata
Match & Map:• Match/Mapping of community
metadata within the Metadata Catalogue
APIs• Definition of APIs for data discovery
and for data access/retrieval