Metadata Harvesting and Validation

Bram VandeputteK.U.Leuven

1

slideshare

• http://www.slideshare.net/bramvandeputte

2

• Validation Service• Integration• Online Validation Service• Harvesting Component

3

Overview

• Interoperability : Application Profile (AP)

• Manual check : very time consuming

• Need a tool for enforcing an AP => validation scheme

• A set of validation rules

• Reusable & extendable

4

Validation Service

Best practices derived from previous projects such as MELT and MACE

Reusable : modular + inheritance possible

• Components :

• XML schema : structure• schematron :

• mandatory/conditional elements

• empty fields

• vocabularies (auto generated)

• ...

• Vcard component

5

Validation Service

• Terminology :

• Validation Component

• Validation Scheme

• Validation Scheme URI :• http://aspect-project.org/validation/ASPECTv1.0/core

6

Validation Service

component : atomic block which does specific validation checking

scheme : collection of components that ensures validity against a whole AP

URI : unique identifier of a scheme

7

Validation Service

8

Validation Service

LOM loose

lomloose.xsd

vcard validator

empty attribute fields

ASPECTv1.0/core

vocabulary bank

Legend

uses

extends

ASPECT

vcard validator

validationScheme

validation component

recommended schematron rules

core schematron rules

ASPECTv1.0/recommended

IMS ILOX

9

Validation Service

!

Integration

• When do we validate ?• Possible integration points :

• Mapping• Harvesting• Enriching

10

11

Online Validation Service demo

12

validation to lre APrefer to lre ap document

13

14

invalid : discarded or identifier recorded for next harvesting

14

The Harvest component

invalid : discarded or identifier recorded for next harvesting

ARIADNE Harvester

harvester log

14

invalid : discarded or identifier recorded for next harvesting

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

14

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

14

LOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

14

LOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

14

LOMLOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

14

LOM

LOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

Validation Msg

14

LOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

Validation Msg

14

LOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

Validation Msg

14

LOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

Validation Msg

14

LOM

SQISPI

invalid : discarded or identifier recorded for next harvesting

OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester

validation serviceharvester log

ASPECT Repository

Validation Msg

15

LOM

LOM

SQISPI

LOMLOM

OAI-PMH

LOM

Validation Msg

LOM

LOM

Validation Msg

1 2

34

5

6

Validation Reports

• After harvesting -> report generated and put online

• report has 4 “levels” :

• full log (incl. metadata)

• reporting log

• Grouped Errors

• Error Summary

16

• Questions ?

17

• SPI : http://ariadne.cs.kuleuven.be/lomi/index.php/SimplePublishingInterface

• IEEE LOM : http://ltsc.ieee.org/wg12/

• OAI-PMH : http://www.openarchives.org/

• Online Validation : http://ariadne.cs.kuleuven.be/validationService/

References

18