Ariadne Harvesting

transcript

ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb

HARVESTING METADATA IN REPOSITORY

NETWORKS

Bram VandeputteK.U.LeuvenARIADNE

• Working since 2006 with Erik Duval

• ARIADNE harvester, OAI-PMH target support, validation service, etc.

• International projects : MACE, MELT, Globe, Ariadne, Aspect, iCoper,...

• Other interests : Information visualisation, Interoperability, TEL

WHO AM I

• OAI-PMH

• Validation Service

• Harvester Infrastructure

• Projects

• MACE

• MELT

• GLOBE

• OAI-PMH Target3

Overview

• Server - Client model

• Pull mechanism

• options :

• selective harvesting

• incremental harvesting

OAI-PMH

• Verbs : Identify, ListRecords, GetRecord

• Parameters :

• baseUrl

• from & until date

• metadataPrefix

• sets

OAI-PMH

• Interoperability : Application Profile (AP)

• Manual check : very time consuming

• Need a tool for enforcing an AP => validation scheme

• A set of validation rules

• Reusable

Validation Service

Validation ServiceLOM loose

MELTv1.0/minimal

MELTv1.0/full

lomloose.xsd

conditional fields check

LRE taxon field numerical

vcard validator

empty attribute fields

MACEv1.0

classification vocab service

melt.xsd mace.xsd

Legend

extends

MACEv1.0

vcard validator

validationScheme

validation component

The ARIADNE Harvester Infrastructure

ARIADNE Harvester

harvester log

ARIADNE Harvester

validation serviceharvester log

ARIADNE Repository

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

LOMLOM

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

Validation Msg

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

Validation Msg

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

Validation Msg

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

Validation Msg

SQISPI

External Repository

LOMLOM

OAI-PMH

LOMLOM

OAI-PMH

ARIADNE Harvester

ARIADNE Repository

Validation Msg

SQISPI

LOMLOM

OAI-PMH

Validation Msg

• MACE

• MELT

• GLOBE

Harvesting in networks

• Metadata for Architectural Contents in Europe

• www.mace-project.eu

• Harvesting, Searching, Enriching, (geo-) Tagging, attention metadata

Harvestedmetadata

DYNAMO

ARIADNE Harvester

OAI-PMH

Enrichedmetadata

MACEmetadata

Uses Lucene for index search

CuminCAD

OAI-PMH

Validation service

• MELT : a Metadata Ecology for Learning and Teaching

• Ministries of education across europe

• enriching, translating, better quality metadata

• practical test for scalability of harvester framework

ARIADNE Harvester

Validation service

EDUCATIO

INDIRE

UL-FRI

ARIADNEMETADATASTORE

SQIARIADNE Harvester

EUN MDS

Enrichment Toolkit

Multilingual Thesaurus

Vocabulary Registry

EUN AMG

• Global Learning Objects Brokered Exchange

• Worldwide network of networks of repositories

• => Both SQI and OAI-PMH are used on various levels in the network

GLOBEMEC

ARIADNE Harvester

EDUCATIO

INDIRE

UL-FRI

DYNAMOARIADNE Harvester

CuminCAD

Federated

Search

EdNA CGIAR Pubelo Merlot

III COSL LORNET Ariadne

LACLO NIME EUN

ISKME KERIS

SQI UDDI

Registry

SQI SQI SQI SQI

SQI Client

ARIADNE Harvester

OAI-PMH target

OAI-PMH result

DYNAMO LOM metadataDYNAMO LOM

metadataDYNAMO LOM metadataRepository LOM

metadata

ARIADNEHarvester

1. Get metadata

out of database

Repository metadata

- identifier- title- url- project description- ...

Repository LOM metadata

- lom.general.identifier- lom.general.title- lom.general.description- lom.technical.location- ...

2. Mapping Process

4. Serve results

3. Copy results in

OAI-PMH service

OAI-PMH Target Software

Repository

• Questions ?

どうも　ありがとうございます！

• SPI : http://ariadne.cs.kuleuven.be/lomi/index.php/SimplePublishingInterface

• SQI : http://ariadne.cs.kuleuven.be/lomi/index.php/LorInteroperability

• IEEE LOM : http://ltsc.ieee.org/wg12/

• OAI-PMH : http://www.openarchives.org/

• OAI-PMH target wiki :http://ariadne.cs.kuleuven.be/lomi/index.php/Setting_Up_OAI-PMH

References

Ariadne Harvesting

Education