V-GISC/SIMDAT project – a Virtual GISC

SIMDAT TMB, 15 December 2004 AMD-1 SIMDAT

V-GISC/SIMDAT project – a Virtual GISCAlfred Hofstadler, Matteo Dell’Acqua

ECMWF


Project History: May 2002: Thirteenth session WMO Regional Association VI: “…agreed that the

concept of a Virtual GISC had merit…”

June 2002: V-GISC in RA-VI Kick-off Meeting Partners: DWD, Meteo France, UK Met-Office, EUMETSAT, ECMWF Steering Group + 4 working groups: Policy, Data, Communications,

Dissemination/Acquisition

2003: SIMDAT project proposal submitted to EU

1 September 2004: contract with EU is signed

October 2004: V-GISC steering group decides to move V-GISC development into the SIMDAT project

November 2004: SIMDAT Kick-off meeting 4 V-GISC working groups are mapped onto SIMDAT working groups: Virtual Organisation,

Ontologies, GRID Infrastructure, Access to Distributed Data

February 2005: First (V-)GISC-demonstrator at CBS


SIMDAT - Introduction Data Grids for Process and Product Development using Numerical Simulation

and Knowledge Discovery

4 years project funded by the EU Contract with EU was signed on 1 September 2004

SIMDAT focuses on 4 applications Product design in automotive and aerospace Process design in pharmacology Service provision in meteorology

Objective of SIMDAT is to use data grid technology to resolve a complex problem for each of the 4 applications

Budget of 11 M € of which 10.5% for meteorological activity 320 men/month taking into account EU funding and the contribution from the

partners


SIMDAT - Strategy 7 Grid-technology areas have been identified to achieving SIMDAT

objectives

Phase 1: Connectivity Phase 2: Interoperability Phase 3: Knowledge

. Deployment of Grid infrastructure with particular attention to data transport and management. Distributed DB access

. Virtual Data Repository

. Introduction of grid technologies research

Workflows for next-generation aggregated knowledge capture, discovery and mining

Ontologies Integration of analysis services Workflows Knowledge Services

Integrated Grid infrastructure offering basic services to applications

Access to data distributed on Grid sites

Management of Virtual Organisation


Meteorology application : Project Aims 5 partners: DWD, Meteo-France, UK Met Office, EUMETSAT and ECMWF

3 “potential” GISCs : DWD, Meteo-France, UK Met Office 2 DCPCs : ECMWF, EUMETSAT

Instead of each National Met Service having a GISC (Global Information System Centre)

The V-GISC will be seen as a normal GISC and will fulfil the WMO Information System technical requirements

The project will build the foundations of the V-GISC by developing an infrastructure that brings together the data of the partners and provides access to the distributed meteorological databases


Meteorology application : Project Aims A complex problem: To build a Virtual GISC, an integrated and scalable

framework for the collection and sharing of distributed data that will offer: A single view of meteorological information which is distributed amongst the 5

partners Improve visibility and access to meteorological data through a comprehensive

discovery service based on metadata development Offer a variety of reliable reliable delivery services (routine dissemination of and

collection of data) Provide a global access control policy managed by the partners and integrated into

their existing security infrastructure Quality of services, reliability and security Processing services and shared data manipulation facilities

The software developed within the project will be made available to WMO


GRID Technology Grid technology will be used

To connect the diverse data sources and create a Virtual Database To enable flexible, secure collaboration through virtual organisation

Data Grid technology presents an architectural framework that aims to provide access to distributed data in a simple,secure, reliable and scalable manner from a widely distributed set of computers and across various administrative boundaries

The essential characteristics of a Data Grid are: Reference a dataset by a unique identifier Discover dataset by attributes Track multiple copies of a single file, and ultimately locate the "nearest" copy Move files from one point on the grid to another point (push, pull and third

party copy) The domain of the V-GISC is an ideal candidate to exploit such a

framework


V-GISC infrastructure

Interface to offer a single view of the data - Discovery facilities - Request/Subscription

MonitoringLoggingControl

Error trackingSecurity

AuthenticationAuthorization

AuditManagementUser registration

DB adminCatalogue admin

Grid infrastructure for sharing data

Interoperability interfaces for data/metadata exchange

mechanisms to synchronise metadata

Dissemination/acquisition mechanisms


Meteo requirements


V-GISC Conceptual view

Through the Distributed Portal users searches for and retrieves data, subscribe to services subject to authentication and authorization

The Virtual Database Service provides a single view of partners databases


V-GISC Conceptual view Virtual Database

Provide the unified view of all the shared datasets through a distributed catalogue Maintain the distributed catalogue amongst the partners using synchronization mechanisms Provide interfaces with the legacy databases Implement data replication mechanisms Preserve the integrity of the data

Access Facilities Collection & Dissemination services that support secure, efficient and reliable transport

mechanisms Quality of Service (QoS): Traffic Prioritization, Queuing mechanisms, Scheduling Discovery service by browsing the catalogue or using a keyword search engine Interactive and batch interfaces

VO Security Services (CA, AuthN, AuthZ, Audit,…) Users management Data policy management Monitoring and control


V-GISC Distributed Architecture V-GISC node is installed on each partner site

All the nodes are interconnected through a dedicated secure communication channel; The Database Communication Layer (DCL)

All the nodes exchange messages through the DCL

The architecture is decentralized No central point where all the nodes are declared No single point of failure

The network of nodes is self-organized The network dynamically accepts new nodes and is aware of node disconnections The network organizes its topology and indicates to the entering new nodes their

position within the network No manual intervention on the nodes to accepts new peers


V-GISC Distributed Architecture


V-GISC Node Each node maintains a copy of the global catalogue describing data

available through the V-GISC The catalogue synchronization is done using the DCL

Each node maintains a cache used to replicate data and to efficiently serve the users

A node is interfaced with the local legacy databases A node has a Web Portal for interactive access A node has a Grid/Web Service Portal for batch access and integration

of the V-GISC in a bigger Grid A node implement all services offered by the V-GISC


V-GISC Node - Functional Design


Demonstrator – Functional View

To deploy a flexible infrastructure on top of which the Virtual Information Centre can be built

To use Grid technologies to federate databases located on partners site

To show to the user a unique view of data sets stored by at least 3 partners

To get a first implementation of the catalogue based on WMO core metadata

To offer first VO security services


Demonstrator - Design 3 main components to build the virtual database: Data Repository,

Catalogue Node and Portal installed on each partner site and interconnected through a dedicated secure

connection channel Data Repository

Interface to the partners databases Offers metadata information to describe, search, locate data Offers interface to retrieve data from the associated local databases

Catalogue Node Maintains the catalogue and ensures synchronisation Harvests metadata and requests data from the data Repository Ingests data and maintains the cache of the V-GISC Serves clients: Portal or other Nodes Monitors the execution of the requests

Distributed Portal Offers interface to search/browse the V-GISC catalogue


Demonstrator - Architectural Choices

Grid Architecture that can accept any kind of Grid Technology Free to choose any grid middleware (OGSA-DAI, GRIA, Glite, GT4) and pick the best

component of each middleware that meets the V-GISC requirement

Catalogue Node built on a J2EE component framework Solid framework used in production environment Includes different services such as persistency, monitoring, configuration, etc The framework can be seen as a kernel of components where it is easy to add services

such as Grid services or Web services

Catalogue duplicated and synchronized on each site To have a fast discovery (browse & search phase) phase To have a reliable system (client redirection to another node in case of problems)


Demonstrator - Architecture


Demonstrator - Deployment


Problems and lessons learned - 1 Grid Middleware

Technology not mature for production environment Middleware still evolving toward standards (WSRF, WSI, …)

Access to distributed data No efficient and robust transport mechanism No mechanism to duplicate and synchronize data Difficult to ensure data integrity on huge data volumes OGSA-DAI is promising, easy to understand and use


Problems and lessons learned - 2 Ontology / Metadata

Meteorological metadata are described using XML WMO-CORE metadata Profile

• Metadata description larger than the data • Same information repeated in all metadata records Unnecessary

information is circulating over the network• Large metadata records slowing down the Database hosting the

catalogue Universal request language was not a solution to the virtual database

problem VO

No standard tools to manage users and data policies No standard security policies


What’s next Finalise the Connectivity phase (by M18/Mar 2006)

Connect EUMETSAT to the Grid (M12-M15/Sep-Dec 2005) Enhance the architecture (M13-M18/Oct 2005-Mar 2006) Implement Registration Authority (M16-M17/Jan-Feb 2006) Improve metadata model (M13-M16/Oct 2005-Jan 2006) Enhance distributed portal (M14-M16/Nov 2005-Jan 2006)

Introduce acquisition of data (M18-M24/Mar-Sep 2006) Develop subscription service (M20-M28/May 2006-Jan 2007) Start developing the Virtual Organisation

Monitoring and management of the system (M18-M24/Mar-Sep 2006) User management and data access control (M24-M30/Sep 2006-Mar 2007)

Develop the discovery mechanism (M20-M25/May-Oct 2006) Start testing with other potential GISC

Japan and Australia have expressed interest in joining the SIMDAT project


Global View : Coordination Effort Metadata Request-reply mechanism Exchange of catalogues Definition on what data should be available and to whom Virtual Organisation Standardisation of services Quality of Service Security

Date post:	25-Feb-2016
Category:	Documents
Upload:	zulema
View:	39 times
Download:	2 times

V-GISC/SIMDAT project – a Virtual GISC

Documents