Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | alvin-riley |
View: | 22 times |
Download: | 1 times |
Interoperability Benchmarking. October 10th 2005 1 © Raúl García-Castro
Benchmarking the Interoperability of Ontology Development Tools
Raúl García-Castro
October 10th, 2005
InteroperabilityWorking DaysOctober 10th-11th, 2005
Interoperability Benchmarking. October 10th 2005 2 © Raúl García-Castro
Ontologies are available in internet
Import
Evaluate
Prune
Merge
Alignment
Identify Differences
Specialize
Extend
Evolution
Export
Conceptualiz
e
Document
Integrate
Anotate
Reasoning
Specify
¿=?
+
O1O2
O3
...
Interoperability Benchmarking. October 10th 2005 3 © Raúl García-Castro
Interoperability problem
• It appears due to ontology reuse.
Tool 1
Tool 2
Tool 3
Tool 4
Tool 5
Ontology development tools
Potential functionalities
Real functionalities
Interoperability Benchmarking. October 10th 2005 4 © Raúl García-Castro
The great diversity in languages and tools originates two problems:– The translation problem:
• How can we translate an ontology between two different languages without losing knowledge?
– The interoperability problem: • How can two tools interchange ontologies or parts of them
without losing knowledge?• How can a tool use ontologies or parts of them from other tool?
Interoperability problem
Why is it difficult?• Different KR formalisms
frames description logicsconceptual graphsfirst order logic semantic networks
• Different modelling components inside the same KR formalism
Interoperability Benchmarking. October 10th 2005 5 © Raúl García-Castro
Interoperability problem
• Some results:– It is difficult to preserve the semantics and the
intended meaning of the ontology– Interoperability decisions…
• At many different levels
• Usually hidden in the programming code of ontology exporters/importers
O. Corcho. A Layered Declarative Approach to Ontology Translation with Knowledge PreservationFrontiers in Artificial Intelligence and Applications, Volume 116, January 2005
Interoperability Benchmarking. October 10th 2005 6 © Raúl García-Castro
Do Tools interoperate?
RDF(S)OWL
...
Ontology development tools
Loom
OILOWL
Ontolingua
RDF(S)
OWL
Query systems
Flogic
DAML+OIL
RQLRDQLSeRQL
SPARQL
Loom
Loom Classifier
OntoBrokerFlora, Flora2
FORID
SilRIRIL
TRIPLE
OIL
FACTRACER
FACTRACERTRIPLE
BOR
DAML+ OIL
FACTRACERTRIPLE
JTP
DQL
OWL-QL
Flora, Flora2
Reasoners
DAML+OIL
ICS-FORTH Validating RDF Parser,RDF Validation Service
DAML Validator,DAML+OIL Ontology Checker
OWL Ontology Validator,OWL Validator
Evaluation tools
Flogic
OWL
OWL
RDF(S)
RDF(S)
RDF(S)
Inter
oper
abili
ty
is cr
ucial
in th
e sem
antic
web
Interoperability Benchmarking. October 10th 2005 7 © Raúl García-Castro
Interoperability approaches
Alternative 1. Ontology interchange using an interchange language Tool i
Interchange language
Tool j
Alternative 2. Direct interchange between each two
Interoperability through application program interfaces (APIs)
Common API
T 3T 2T 1
T 6T 5T 4
Alternative 3. Common API
Interoperability through an interchange language
T 3
T 2T 1
T 4
T 5 T 6
Interoperability Benchmarking. October 10th 2005 8 © Raúl García-Castro
Knowledge models comparison
Classes
Template Slots/properties/instance attributes
InstancesData types
RDF(S)
Subclass-of
Subproperty-of
Literals
Containers
Collections
Statements
Concept groups
Disjoint decompositions
Exhaustive decompositions
Partitions
Constants
Relation properties
Synonyms
Abbreviations
Bibliographic references
Metaclasses
Own slots
Class attributes
Interoperability Benchmarking. October 10th 2005 9 © Raúl García-Castro
Translation strategies
Thesis
MSc Thesis PhD Thesis
Disjoint-subclass
Partial loss
Total lossDon’t export
Insert ad-hoc RDF(S)
<rdfsClass rdf:about=“#Thesis”> <a:disjoint rdf:Resource=“#MsC Thesis”> <a:disjoint rdf:Resource=“#PhD Thesis”></rdfs:Class>
Doesn’t import
Thesis
RDF(S) IMPORTEXPORT
Thesis
MSc Thesis PhD Thesis
subclass
Minimize knowledge loss
Thesis
MSc Thesis PhD Thesis
subclass
Thesis
Interoperability Benchmarking. October 10th 2005 10 © Raúl García-Castro
Interoperability using an interchange language
Interoperability is not guaranteed with the existence of importers and exporters
from the tools to the interchange language.
Interoperability using an interchange language depends on the correct working of
the translators…
…and these translators don’t work properly.
RDF(S)OWL
... Loom
OILOWL
Ontolingua
RDF(S)
Interoperability Benchmarking. October 10th 2005 11 © Raúl García-Castro
How can we improve this situation?
Goal 1:• To assess and improve the interoperability of ontology development tools using
RDF(S) for ontology exchange.
Goal 2:• To identify the subset of RDF(S) elements that ontology development
tools can use to correctly interoperate.
Goal 3:• Next step: OWL.
Benchmarking: • Evaluation of the interoperability• Improvement of the interoperability• Collection of best practices
Interoperability Benchmarking. October 10th 2005 12 © Raúl García-Castro
General framework for benchmarkingBENCHMARKING ITERATION
Recalibration task
PLAN PHASE
1. B. goals identification 2. B. subject identification 3. Participant identification 4. B. proposal writing 5. Management involvement 6. B. partner selection 7. B. planning and resource allocation
IMPROVE PHASE
11. B. report writing 12. B. findings communication 13. Improvement planning 14. Improvement 15. Monitor
EXPERIMENT PHASE
8. Experiment definition 9. Experiment execution 10. Experiment results analysis
General evaluation criteria:• Interoperability• Scalability• Robustness
Benchmark suites for:• Interoperability• Scalability• Robustness
Benchmarking supporting tools:• Testing frameworks• Workload generators• Monitoring tools• Statistical packages
García-Castro, Maynard, Wache, Foxvog and González-Cabero. Knowledge Web Deliverable 2.1.4 Specification of a methodology, general criteria, and benchmark suites for benchmarking ontology tools. December 2004.
Interoperability Benchmarking. October 10th 2005 13 © Raúl García-Castro
Plan phase
Benchmarkinggoals
identification
Benchmarkingsubject
identification
Participantidentification
Need for benchmarking
Organisation goals and strategies Benchmarking
proposal writing
Benchmarking goals, benefits, costs
Benchmarking subject, tool functionalities, evaluation criteria
List of involved members, benchmarking team
Management involvement
Benchmarking partner selection
Benchmarking planning and
resource allocation
Benchmarking proposal
Management support
Benchmarking partners, updated benchmarking proposal
Benchmarking planning
Organisation's tools
Tools from outside the organisation
Organisation planning
Improve the interoperability of ontol. development tools
RDF(S) import and export capabilities
Identify ontology components
exported/imported
B.P.
Interoperability Benchmarking. October 10th 2005 14 © Raúl García-Castro
Experimentdefinition
Experimentexecution
Experimentanalysis
Benchmarking planning
Benchmarking proposal Experiment
definition, experimentation planning
Experiment results
Experiment report
Experiment phase
RDF(S) Import benchmark suitesRDF(S) Export benchmark suites
• test 1• test 2• test 3• ...
• test 1• test 2• test 3• ...
NOOKOK
• test 1• test 2• test 3• ...
OKNOOK
OKOKNO ...
• test 1• test 2• test 3• ...
• test 1• test 2• test 3• ...
NOOKOK
• test 1• test 2• test 3• ...
OKNOOK
OKOKNO
E.R.
Interoperability Benchmarking. October 10th 2005 15 © Raúl García-Castro
Benchmarkingreport writing
Benchmarking findings
communication
Improvementplanning
Updated benchmarking proposal
Experiment report
Improvement
Benchmarking report
Updated benchmarking report
Monitor
Necessary changes, improvement planning, improvement forecast
Organisation support
Improved tool
Monitorisation report
Improve phase
• Comparative analysis• Compliance with standards• Weaknesses• Recommendations on tools• Recommendations on practices
Interoperability Benchmarking. October 10th 2005 16 © Raúl García-Castro
Identification of the evaluation elements and metrics
GQM paradigm: Any software measurement activity should be preceded by:
1.- The identification of a software engineering goal ...
2.- ... which leads to questions ...
3.- ... which in turn lead to actual metrics.
To improve the interoperability of ontology development tools using RDF(S) as an interchange language
Q1: Which are the elements of the knowledge model of a tool that are interchanged with another tool without knowledge loss?
Q2: Which are the elements of the knowledge model of a tool that are interchanged with another tool with knowledge loss?
Q3: Which are the elements of the knowledge model of a tool that are not interchanged with another tool?
M1: Elements of the knowledge model of a tool that are exported to RDF(S)
M5: Elements of the knowledge model of a tool that are not exported to RDF(S)
M2: Knowledge loss when exporting to RDF(S) an element of the knowledge model of a tool
M3: Elements of the knowledge model of a tool that are imported from RDF(S) intro another tool
M6: Elements of the knowledge model of a tool that are not imported from RDF(S) intro another tool
M4: Knowledge loss when importing from RDF(S) into another tool an element of the knowledge model of a tool
Interoperability Benchmarking. October 10th 2005 17 © Raúl García-Castro
Evaluation infrastructure
Ontology Comparer
Ontology Loader
Ontology Development
Tool
Ontology Storer
Import Benchmark
Suite
Export Benchmark
Suite
RDF(S) Comparer
Import Benchmark
Suite Executor
Export Benchmark
Suite Executor
Ontology Repository
RDF(S)File
Repository
Export experiments
Import experiments
Interoperability Benchmarking. October 10th 2005 18 © Raúl García-Castro
Export experiments
Ontology Loader
Ontology Development
Tool
Export Benchmark
Suite
RDF(S) Comparer
Ontology Repository
RDF(S)File
Repository
Concept 1
Concept 2 Concept 3 Concept 4
= ?YES
NO
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class></rdf:RDF>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class></rdf:RDF>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class></rdf:RDF>
Interoperability Benchmarking. October 10th 2005 19 © Raúl García-Castro
Import experimentsOntology Comparer
Ontology Development
Tool
Ontology Storer
Import Benchmark
Suite
Ontology Repository
RDF(S)File
Repository
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class></rdf:RDF>
Concept 1
Concept 2 Concept 3 Concept 4
YESConcept 1
Concept 2 Concept 3 Concept 4= ?
Concept 1
Concept 2 Concept 3 Concept 4 NO
Interoperability Benchmarking. October 10th 2005 20 © Raúl García-Castro
Experimentation in the benchmarking
• Agreement phase. The quality of the benchmark suites is essential for the results.
• Evaluation phase 1. The RDF(S) importers and exporters of the ontology development tools are be evaluated.
• Evaluation phase 2. The ontology exchange between ontology development tools is evaluated.
WE ARE HERE
Interoperability Benchmarking. October 10th 2005 21 © Raúl García-Castro
Benchmarking the Interoperability of Ontology Development Tools
Raúl García-Castro
October 10th, 2005
InteroperabilityWorking DaysOctober 10th-11th, 2005