Journal of Digital Information Management q Volume 6 Number 5 q October 2008366

The OntoNL Semantic Relatedness Measure for OWL Ontologies

Anastasia Karanastasi, Stavros ChristodoulakisLab. Of Distributed Multimedia Information Systems / Technical University of Crete (MUSIC/TUC) University Campus, Kounoupidiana, Chania, Greece [allegra, stavros]@ced.tuc.gr

Journal of Digital Information Management

AbstrAct: An effect of the growing importance of the Semantic Web used for sharing knowledge over the Internet was the development and publishing of many ontologies in different domains. This led to the need of developing mechanisms for capturing the semantics of the ontologies. In this paper, we introduce the OntoNL Semantic Relatedness Measure, a fully automated way of measuring, in an asymmetric way, semantic relatedness between concepts of domain ontologies. We have developed metrics to guide the automation of the procedure by using feedback from an extensive evaluation with human subjects. We conclude with an application based evaluation using the OntoNL Framework as the application platform, a Natural Language Interface Generator for Knowledge Repositories.

Categories and Subject DescriptorsI.2.7 [Natural Language Processing]: I.2 [Artificial Intelligence]; Natural language interfaces General terms: Semantic web, Natural Language Interface GeneratorKeywords: OntoNL Framework, Domain ontology, Knowledge Repositories

1. Introduction

The Semantic Web technologies have started to make a difference in making content machine processable and have begun to creep into use in some parts of the World Wide Web. This is accomplished by the use of ontologies that describe context in different domains. An ontology describes a hierarchy of concepts usually related by subsumption relationships. In more sophisti-cated cases, suitable axioms are added in order to express other relationships between concepts and to constrain their intended interpretation (Guarino 1998). A module dealing with ontologies can perform automated reasoning using the ontologies, and thus provide advanced services to intelligent applications such as: conceptual/semantic search and retrieval, software agents, decision support, speech and natural language understanding and know-ledge management. The need to determine semantic relatedness between two lexically expressed concepts is a problem that con-cerns especially natural language processing. Measures of relatedness or distance are used in applications of natural language processing as word sense disambiguation, determining the structure of texts, information extraction and retrieval and automatic indexing. The methodology for calculating the semantic Relatedness of the concepts of a domain ontology is integrated in the OntoNL Framework (Karanastasi 2006), a natural language interface generator to knowledge repositories.

Given an OWL ontology, weights are assigned to links between concepts based on certain properties of the on-tology, so that they measure the level of relatedness between concepts. In this way we can identify related con-cepts in the ontology that guide the semantic search procedure. An important property of the OntoNL Semantic Relatedness measure is that it is asymmetric (the relatedness between A and B does not imply the opposite) since relations that are described with natural language do not indicate mathematical rules.The NLP literature provides the largest group of related work for measuring semantic relatedness that in most cases are based on lexical resources or WordNet (Fellbaum 1998) and other semantic networks or deal with computing taxonomic path length.All the research results presented in the literature so far (Resnik 1995, Rada 1998, Jiang 1997, Budanitsky 2005, Jarmasz 2003) were tested on specific ontologies like the WordNet and MeSH ontologies, they are not gen-eral and have not been tested in different domain ontologies that refer to different contexts. The WordNet and MeSH ontologies are well formed hierarchies of terms and the methodologies that have used them examined basically similarity between terms and not relatedness between concepts. Also, most of these approaches are focused on the comparison of nouns, limiting their generality to complex objects or even hierarchies of verbs.In this paper we present the automation of the procedure for calculating the semantic relatedness between concepts of domain ontologies by using extensive experimentation with human subjects to fine tune the parame-ters of the system and to evaluate the performance of the OntoNL Semantic Relatedness Measure in different do-mains with different domain ontologies.

2. The OntoNL Semantic Relatedness Measure

The OntoNL Semantic Relatedness Measure depends on the semantic relations defined by OWL vocabulary. The methodology borrows and expands ideas from the research of Semantic Relatedness of concepts in semantic networks and can be found in details in (Karanastasi 2007a). The algorithm takes into account the semantic relation of OWL: EquivalentClass. The class that is OWL: Equiva-lentClass with a source class has a similarity (not relatedness) value 1. In our computations, the classes related to the source class of the ontology are also related with the same value to the equivalent class. We count the number of the common properties the two concepts share (numerator) and divide it with the number of the initial concept (denominator) and the number of the common properties the two concepts share that are inverseOf properties (numerator) and divide it with the number of the common properties the two concepts share (denominator):

Journal of Digital Information Management q Volume 6 Number 5 q October 2008 367

1 2 2 1 2

1 11 2 1 2

1 1

, 0, 1:

( , ) ( ) ( ),

n n

ijk invijki i

prop n n

ij ijki i

f f f f f

p prel c c f f

p p

= =

= =

∀ ≥ > + =

= × + ×∑ ∑

∑ ∑ (1)

In (1), the value pij represents the fact that concept cj is related to concept ci (value: 0 or 1 in general). The value pijk represents the fact that both concepts cj and ck are related to concept ci. The pinvijk represents the fact that both concepts are inversely related. The factors f1 and f2 in general depend on the ontologies used, and we assume that they are experimentally determined for a given ontology.The conceptual distance measure is based on three factors; the path distance, the specificity and the specialization. The path distance measures the relatedness of two concepts by counting the minimal path of edges between the two concepts through their structural relations (IS-A relations):

1 2 1 2

1 21 2

1, 1, 2 :

( , ) (0,1]2

C C C C

C C

d d d dd dpathDist c c

D

∀ ≥ ≥ + >

+= ∈

∗ (2)where dC1 is the number of edges from concept 1 to the closer common subsumer and dC2 the number of edges from concept 2 to the closer common subsumer. D is the maximum depth of the ontology.We claim that when the change of direction (from superClassing to subClassing and opposite) is close to the concept/subject of the language model (dC1 << (dC1+dC2)/2), the two concepts are more related. When the direction of the path changes far from the reference concept then the semantics change as well (more specialization of the reference concept c1 in comparison with the subsumer concept).We count the specificity of the concepts inside the ontology by the following normalized weight value:

1 2 11 1

1 2

2: _1 ln (0,1]2

C C CC specC

C C

d d dd wd d

+ ×< = − ∈

+

then 1 2

1 1: _1 02

C CC specC

d dd w+≥ =

(3)

We, also propose a method of counting the specialization of the concept - C1 based on the object properties of the subsumer, by the factor:

1

1

1

# 10 # :#_ 2 1 log [0,1]#C

C S

Cspec

S

ObjP ObjPObjPwObjP

∀ ≤ ×

= − ∈ (4)

where ObjPC1 is the number of Object Properties of the concept C1 and ObjPS is the number of ObjectProper-ties of the subsumer concept. The conceptual distance measure then becomes:

1 1 1 2_1 _ 2 1 ( , )3

specC specCCD

w w pathDist c crel

+ + −=

(5)

The related senses measure counts the common senses of two concepts by counting the common nouns and synonyms extracted from the descriptions of the concepts in the ontology (owl:label, owl:comment) or from the descriptive part of the term meaning in the WordNet. Let S1 be the description set of nouns for c1 and S2 the description set of nouns for c2. The related senses measure is:

1 21 2

1 2 1 2

( , )| | \RS

S Srel c c

S S S S∩

=∩ + (6)

The overall relatedness measure is the following:

1 2 3 1 2 3

1 2 1 2 1 2

1, ( , , ) 0,( , ), ( , ), ( , ) [0,1] :PROP CD RS

w w w w w wrel c c rel c c rel c c∀ + + = >

∈

1 2 3OntoNL PROP CD RSrel w rel w rel w rel= × + × + ×

The three factors w1, w2 and w3, help of balancing among the parameters depending on the application ontol-ogy.

3. Experimental Evaluation

We have focused our attention to the performance experimentation in a generic way utilizing readily available ontologies in the web, not carefully constructed by hand ontologies. As we discussed in the previous section the three factors w1, w2 and w3 of the overall OntoNL measure help of balancing among the three sub-measure depending on the application ontology. We need to bound their values and provide the complete measurement that will show good results regardless of the OWL ontology used. In order to assess the impact of each of the sub-measures we needed to evaluate it against a “gold standard” of object relatedness. To that end we designed a detailed experiment in which human subjects, selected from the Liberal Arts field and Computer Science field, were asked to assess the relatedness between pairs of objects (the results and discussion over the results can be found in (Karanastasi 2007a). The object pairs were selected from a number of ontologies freely available in the web1. The selection of the ontologies was based on the public availability of the ontologies and by the subjects’ ability to relate to the ontology content (domain).Our first objective was to investigate what are the values of the parameters f1, f2, w1, w2, w3 for each ontology, and overall. We observed that the best computed manually values of these parameters strongly depend on the ontology. Their “optimal” experimental values are shown in Table 1.

Ontology relPROP relOntoNL

f1 f2 w1 w2 w3Soccer Ontology 0,5 0,5 0,7 0,2 0,1

Wine Ontology 0,65 0,35 0,5 0,25 0,25People Ontology 0,1 0,9 0,45 0,2 0,35

Pizza Ontology 0,65 0,35 0,5 0,27 0,23Koala Ontology 0,99 0,01 0,25 0,65 0,1Images Ontology 0,33 0,67 0,45 0,5 0,05

Travel Ontology 0,9 0,1 0,7 0,1 0,2

Table 1. The values of the relative weights f1 and f2 of (1) and w1 (for relPROP), w2 (for relCD) and w3 (for relRS) of (8) for the test set of ontologies

� Images:http://www.mindswap.org/glapizco/technical.owl Koala:http://protege.stanford.edu/plugins/owl/owl-library/koala.owl People:http://owl.man.ac.uk/2005/07/sssw/people.html Pizza:http://www.co-ode.org/ontologies/pizza/2005/05/�6/pizza.owl Soccer:http://lamia.ced.tuc.gr/ontologies/AVMDS03/soccer Travel:http://learn.tsinghua.edu.cn/home-page/20032�4945/travelontology.owl Wine:http://www.w3.org/TR/owl-guide/wine.rdf

Journal of Digital Information Management q Volume 6 Number 5 q October 2008368

Using the best computed manually values for the parameters we studied how the computed relatedness meas-ure among two concepts was correlated with the relatedness perceived by the human subjects.

Human Subjects RatingsMeasure relPROP relCD relRS relOntoNL

Soccer Ontology 0,910 0,594 0,329 0,943Wine Ontology 0,832 0,644 0,830 0,976People Ontology 0,906 0,937 0,949 0,984Pizza Ontology 0,657 0,77 - 0,863Koala Ontology 0,492 0,846 0,285 0,857Images Ontology 0,964 0,953 0,273 0,997Travel Ontology 0,946 0,891 0,612 0,973

Table 2. The values of the coefficients of correlation between hu-man ratings of relatedness and the OntoNL Semantic Relatedness sub-measures and overall measure

Table 2 shows the computed correlation coefficients with relative weights of Table 1 between the system com-puted relatedness measure and the human subjects evaluated relatedness.

4. The OntoNL Semantic Relatedness Measure’s Weight Value Calculation

An observation mentioned above was the relatively large variability of the optimal weights for each ontology. Our scope was to develop an automatic method for determining the weights for any given ontology. We first determine the features of the OWL Ontology structure that we essentially can state their impact in the OntoNL Semantic Relatedness Measure:

Feature 1: Let C be a set whose elements are called concepts or classes. Let C N∈ were { }: 1, 2,3,...N = , be the number of all Classes of the OWL Domain Ontology.Feature 2: Let P be a set whose elements are called Object Properties. Let P N∈ were { }: 1, 2,3,...N = , be the number of all Object Properties of the OWL Domain Ontology. Feature 3: Let CH be a class hierarchy, a set of classes.

CH is a directed, transitive relation CH C C⊆ × which is also called class taxonomy. ( , )C

s iH C C is the set where Cs is a sub-class of Ci. The number of subclasses (Cs) for a class Ci is defined as ( ),C

s iH C C

Feature 4: A specific kind of relations is attributes A. The function :att A C→ with ( ):range A STRING= relates concepts with literal values.The values of these features can be computed univocally in each case of ontologies we used for the evaluation experiments. The metrics we are proposing are not ‘gold standard’ measures of ontologies. Instead, the metrics are intended to evaluate certain aspects of ontologies and their potential for knowledge representation. To define the metrics we used as a guideline, work on ontology quality analysis (Tartir, 2005, Vrandecic, 2007). The category of metrics we are interested in is the schema metrics that evaluates ontology design and its potential for rich know-ledge representation. Metric 1 (μ1)-Object Property Richness: This metric (PR) is defined as the average number of object proper-ties per class. It is computed as the number properties for all classes (P) divided by the number of classes (C).

PPR

C=

(9)Metric 2 (μ2)-Inverse Object Property Richness: This metric (PRinv) is defined as the average number of in-verse object properties per class. It is computed as the number properties for all classes (Pinv) divided by the number of classes (C).

invinv

PPR

C=

(10)Metric 3 (μ3)-Specificity Richness: This metric (SR) is

defined as the sum of all inner classes i innerC C∈∑

(all classes of the ontology except the leaf classes) of the number of properties of the subclass Cs of a class Ci ( ( ),C

S iH C CP ), minus

the number of properties of the class Ci ( iCP ) divided by the number of properties of the sub-class Cs ( ( ),C

S iH C CP ). This sum

is divided by the number of the inner classes of the ontology.

( )( , )

(( ) ),

j

iC

i j S i

CCC

C Cinner C H C C s i

inner

PP

H C CSR

C∈ ∈

−

=∑ ∑

(11)Metric 4 (μ4)-Inheritance Richness: The inheritance richness of the schema (IR) is defined as the average number of subclasses per class.

( , )i

CS i

C CH C C

IRC

∈=∑

(12)Metric 5 (μ5)-Readability: This metric indicates the existence of human readable descriptions in the ontology, such as comments, labels, or captions. Formally, the readability (R) of a class is defined as the sum of the number attributes that are comments and the number of attributes that are labels the class has.

, : , :R A A rdfs comment A A rdfs label= = + = (13)If the readability is equal to zero, then we define the readability as the average number of classes with one-word string names per all the classes of the ontology.

, : _ _, :

A A rdfs one word IDR

A A rdfs ID=

== (14)

We are going to use methodologies from Linear Programming field (Gartner 2006) so to compute the impact of each metric to the weights of the OntoNL Semantic Relatedness Measure and the results that we have obtained empirically through experimentation as training data to determine the exact weight values of the metrics that we think that affect the parameters (f1, f2, w1, w2, w3) of the OntoNL Semantic Relatedness Measurement. We observe that the best computed manually values of f1, f2, w1, w2, w3 is affected by the characteristics of the ontology structure and description and the ontology metrics defined above. We define:f1 = f1(μ1, μ3) (the influence parameter of the submeasure of the relOP) to indicate that f1 depends on the ontolo-gy metrics μ1 (object property richness) and μ3 (specificity richness). f2 = f2(μ1, μ2, μ3) (the influence parameter of the submeasure of the relOP) is affected by the ontology metrics μ1 (object property richness), μ2 (inverse object property richness) and μ3 (specificity richness).

Journal of Digital Information Management q Volume 6 Number 5 q October 2008 369

w1 = w1(μ1, μ2, μ3) (the influence parameter of relOP) is affected by the ontology metrics μ1 (object property rich-ness), μ2 (inverse object property richness) and μ3 (specificity richness).w2 = w2(μ1, μ3, μ4) (the influence parameter of relCD) is affected by the ontology metrics μ1 (object property rich-ness), μ3 (specificity richness) and μ4 (specificity richness).w3 = w3(μ4, μ5) (the influence parameter of relRS) is affected by the ontology metrics μ4 (specificity richness) and μ5 (readability).We want to determine the how much the metrics affect the influence parameters f1, f2, w1, w2, w3 of the OntoNL Semantic Relatedness Measurement. To that purpose we have computed the ontology metrics for the 7 OWL domain ontologies that we have used for experimentation. Then we defined the objective functions to represent the problem as a linear programming problem. Since we assume a linear dependency of the parameters f1, f2, w1, w2, w3 from the ontology metrics we can write:

In these equations cij are constants and ei’s are error values. As training ontologies we will use the ones that we described above. For each one of these ontologies we have calculated the values of μ1, μ2, μ3, μ4 and μ5. We also used as values for f1, f2, w1, w2, w3 the values that gave the maximum correlations for the concept relatedness in the user experiments (table 1). The seven OWL Domain Ontologies that were used for experimentation were: (1) Soccer Ontology, (2) Wine Ontology, (3) People Ontology, (4) Pizza Ontology, (5) Koala Ontology, (6) Images Ontol-ogy and (7) Travel Ontology.We used a Linear Solver to compute the different c values and the deviations e from the values of Table 1. By calculating the values of the metrics and by multiplying them with the corresponding c values we will get the values of the influence parameters of the OntoNL Semantic Relatedness Measure automatically. The results of the linear programming procedure are presented in Tables 3-7.

The weight values definition problem for f1

Name Final Valuec1 0,758196161c3 0,435593623e11 0,05333553e12 -0,074884789e13 0,130046084e14 -0,086183898e15 -0,002707919e16 -0,004941855e17 -0,014663153

Table 3. The values of the constants of the metrics that influence the weight value f� of the OntoNL Sem. Rel. Measure and the deviations from the ontologies

In table 3 we find the values of c1 and c3 that we will use to multiply the computed ontology metrics μ1 and μ3 respectively in order to define the influence parameter f1 of a domain ontology we want to process. The e11-e17 values are the deviations from the human judgments for each one of the seven ontologies used for experimenta-tion.

The weight values definition problem for f2

Name Final Valuec1 0,049345079c2 0,840100697c3 0,047572988e21 0,136529522e22 0,022606066e23 -0,070313878e24 0,022606066e25 0,058533106e26 -0,148934617e27 -0,021026264

Table 4. The values of the constants of the metrics that influence the weight value f2 of the OntoNL Sem. Re-latedness Measure and the deviations from the ontologies

In table 4 we find the values of c1, c2 and c3 that we will use to multiple the computed ontology metrics μ1, μ2 and μ3 respectively in order to define the influence parameter f2 of a domain ontology we want to process. The e21-e27 values are the deviations from the human judgments for each one of the seven ontologies used for experimentation.

The weight values definition problem for w1

Name Final Valuec1 0,549347616c2 0,310262499c3 0,26487096e31 0,06518881e32 -0,066029702e33 -0,117953097e34 -0,066029702e35 0,021163584e36 -0,267102916e37 -0,061115641

Table 5. The values of the constants of the metrics that influence the weight value w� of the OntoNL Sem. Relatedness Measure and the deviations from the ontologies

In table 5 we find the values of c1, c2 and c3 that we will use to multiple the computed ontology metrics μ1, μ2 and μ3 respectively in order to define the influence parameter w1 of a domain ontology we want to process. The e31-e37 values are the deviations from the human judgments for each one of the seven ontologies used for experimentation.

The weight values definition problem for w2

Name Final Valuec1 0,363197897c3 0,046685606c4 0,245670362e41 0,08867345

Journal of Digital Information Management q Volume 6 Number 5 q October 2008370

e42 0,016971459e43 -0,043044046e44 0,009254977e45 -0,212717517e46 -0,066789378e47 0,207651054

Table 6. The values of the constants of the metrics that influence the weight value w2 of the OntoNL Sem. Relatedness Measure and the deviations from the ontologies

In table 6 we find the values of c1, c3 and c4 that we will use to multiple the computed ontology metrics μ1, μ3 and μ4 respectively in order to define the influence parameter w2 of a domain ontology we want to process. The e41-e47 values are the deviations from the human judgments for each one of the seven ontologies used for experimentation.

The weight values definition problem for w3

Name Final Valuec4 0,278023071c5 0,315776889e51 0,016872301e52 -0,056071167e53 -0,098853789e54 -0,037958858e55 0,064273687e56 0,149284072e57 -0,037546245

Table 7. The values of the constants of the metrics that influence the weight value w3 of the OntoNL Sem. Relatedness Measure and the deviations from the ontologies

In table 7 we find the values of c4 and c5 values that we will use to multiple the computed ontology metrics μ4 and μ5 respectively in order to define the influence parameter w3 of a domain ontology we want to process. The e51-e57 values are the deviations from the human judgments for each one of the seven ontologies used for experimentation. From the results presented in the Tables 3-7 we can see the most important deviations from the empirical results we obtained by experiment with the human subjects. For the factor f1 we get the largest deviation for the ontology People since it is an ontology with a small number of Object Properties in comparison to the Classes that it has.For the factor f2 we get the largest deviations for the ontologies Soccer and Images because of the little number of the inverse Object Properties for the Soccer Ontology and the lack of Specificity Richness as it was defined earlier in the Metrics for the Images ontology.For the factor w1 we get the largest deviations for the ontologies People and Images because of the reasons that influence the bad performance in the calculation of the values of f1 and f2.For the factor w2 we get the largest deviations for the ontologies Koala and Travel because they are quite flat as domain ontologies, they do not have a large Inheritance Richness as it was defined in the Metrics definition.For the factor w3 we get the largest deviation for the ontologies Images because it does not have descriptions, like comments and labels and because the names of the classes are mainly two word strings, so the methodology of finding related senses using the WordNet did not have good results.

In conclusion, the methodology of linear programming helped the determination of an automatic way for calculating the influence parameters (f1, f2, w1, w2, w3) of the OntoNL Semantic Relatedness Measure. The methodology showed that with the correct definition of ontology metrics we get realistic results for the relatedness of concepts of a domain ontology. The deviations from the human judgements were expected if we confront the ontology metrics and the ontologies we used for experimentation. The methodology we used to define those ontology metrics was based on the feedback of the users we used for the experimentation. By using a more systematic way of extracting the knowledge and experience of the users’ maybe could lead to a more accurate definition of ontology metrics with even better results in comparison with human judgements.

4. An Application of the OntoNL Framework

We have implemented an application of the OntoNL Framework that addresses a semantic multimedia repository with digital audiovisual content of soccer events and metadata concerning soccer in general. The overall architecture of the OntoNL Framework is shown in figure 1 (OntoNL Component). The Framework in a particular application environment has to be supplied with domain ontologies (encoded in OWL) which are used for semantic processing. The user input in an application environment is natural language requests, yes/no questions and WH-questions (who, were, what, etc.). The output for a particular natural language input query is a set of one or more weighted disambiguated to the specific domain queries, encoded in SPARQL. We choose SPARQL as the query language to represent the natural language queries since SPARQL is defined in terms of the W3C’s RDF data model and will work for any data source that can be mapped into RDF. If the environment uses a different type of repository than OWL-SPARQL, a module has to be implemented that does the mapping from the SPARQL encoded queries to the schema and query language that the environment uses (Relational Schema-SQL, XML Schema-XQUERY, etc). Since this transformation is Schema dependent it is not automated within the Framework software. The reference ontology we used is an application of the DS-MIRF ontological infrastructure (Tsinaraki 2006) and the WordNet for the syntactic analysis. The repository for accessing the instances is the DS-MIRF Metadata Repository (Tsinaraki 2006). The DS-MIRF OntoNL Manager provides the OntoNL component with the ontologies for the disambiguation and the natural language expression for disambiguation.The OntoNL Component provides the NL Ontology API and the NL Query API for communication. The NL Query API contains functions to input a natural language query and after the disambiguation outputs a number of weighted SPARQL queries, based on the structures of the ontologies used for the disambiguation. It implements functions for the data transfer between the Framework and the repository. The NL Ontology API consists of the total of functions used for manipulating the ontologies that interfere with the system.The DS-MIRF OntoNL Manager provides the OntoNL component with the ontologies for the disambiguation and the natural language expression for disambiguation. It is also responsible for retrieving the user request, communicate with the repository, manage the results, rank them based on any existing User Profile information and presented them to the front end the user uses for interactions.As we already mentioned, the output of the OntoNL is weighted SPARQL queries. To interface with DS-MIRF we had to develop mappings of the SPARQL to the retrieval language of DS-MIRF which intern uses XQuery to access semantic MPEG-7 multimedia content from the XML DBMS.

Journal of Digital Information Management q Volume 6 Number 5 q October 2008 371

Figure 1. The Architectural Representation of the Application using the OntoNL Framework

5. Evaluating the Application

We have performed an application-based evaluation of the OntoNL Semantic Relatedness Measure. The applica-tion used the OWL Ontology for the domain of soccer (http://lamia.ced.tuc.gr/ontologies/AV_MDS03/soccer ), be-cause it is a large and very specific ontology. Also, the context of the ontology is familiar with the users.We first asked the users to submit requests. The requests are of disambiguation type (2) as it is defined in the OntoNL Disambiguation algorithm definition (Karanastasi 2007b).The disambiguation type (2) concerns the case where one of the subject or object part of the language model cannot be disambiguated by using the ontology repository:

ex.1 “… information about soccer team Milan”In this example, the word soccer team is matched to the corresponding concept of the domain ontology, but the system cannot resolve the ambiguity of the subject part of the natural language expression, information. The system considers the word information as an unresolved concept.ex.2 “…the players of Barcelona”In this example, the word players is matched to the corresponding concept of the domain ontology but the system cannot resolve the ambiguity of the object part of the natural language expression, Barcelona. Since the system cannot find a concept or a property of the domain ontology that could be matched to the word Barcelona it “considers” it as a concept instance.

Journal of Digital Information Management q Volume 6 Number 5 q October 2008372

We gathered a total of 60 requests, after eliminating any duplicates. We distinguished the types of expressions based on the OntoNL Language Model in 3 different types:

Subject PartSubject Part – Conjuctive/Disjunctive/Plain Verb PhraseSubject Part – Verb – Conjuctive/Disjunctive/Plain Object Part

We have presented to the human subjects, the resulted concepts related to the subject concept of their request. The users replied the ranking position of their correct response in mind and this experiment was conducted twice. Since our results are a ranked list, we use a scoring metric based on the inverse rank of our results, similar to the idea of Mean and Total Reciprocal Rank scores described in (Radev 2002), which are used widely in evaluation for information retrieval systems with ranked results. Hence our precision and recall are defined as:

1

#rankingPRECISION

requests=∑

( _ )#

n accepted rankingRECALLrequests

=

The precision is depended on the ranking position of the correct related concept to the subject concept of the request. The recall is depended on the number of the related concepts the algorithm returns. In Table 8 we present the precision and recall scores we obtained for the two most complex datasets of request types.

DataSet Preci-sion

Recall (n=3)

Recall (n=5)

Recall (n=8)

Subject Part – Conjuctive/Disjunctive/Plain Verb Phrase (15 requests)

49% 60% 86,7% 100%

Subject Part – Verb – Conjuctive/Disjunctive/Plain Object Part (25 requests)

39,7% 52% 76% 92%

Total 44% 55% 80% 95%

Table 8. Quality metrics for the first iteration

What we see is that overall we gain more than 50% of the correct matches in the first three hits and that the re-quests of type Subject Part – Conjuctive/Disjunctive/Plain Verb Phrase had better precision and recall than the re-quests of type Subject Part – Verb – Conjuctive/Disjunctive/Plain Object Part requests. This is because we use the verbs in this application to disambiguate in a more sufficient way the RelationTypes

•••

modeled in the OWL Domain Ontology for Soccer that is based on the MPEG-7 (Tsinaraki 2004).

After this experiment we asked the users to submit new requests and we once again gathered 20 requests. In Table 9 we present the precision and recall scores we obtained for the two most complex datasets of request types and for a second iteration of the experiment.

DataSet Preci-sion

Recall (n = 3)

Recall (n =5)

Recall (n =7)

Subject Part – Conjuctive/Disjunctive/Plain Verb Phrase (10 re-quests)

47% 70% 90% 100%

Subject Part – Verb – Conjuctive/Disjunctive/Plain Object Part (10 requests)

46,1% 60% 100% -

Total 46,63% 65% 90% 100%

Table 9. Quality metrics for the second iteration

What we see here is that in total we gain a 65% of the correct matches in the first 3 results of the OntoNL Disambiguation Procedure an a 90% in the first 5 results. The overall conclusion that derives is that in a second iteration of tests the performance was better because of the familiarity of the users using the system increased. In more details, the request type Subject Part – Conjuctive/Disjunctive/Plain Verb Phrase has a better precision but the request type Subject Part – Verb – Conjuctive/Disjunctive/Plain Object Part has a better recall.

Figure 2. The precision of the OntoNL measure to the user input for the requests

Journal of Digital Information Management q Volume 6 Number 5 q October 2008 373

Figure 3. The precision of the OntoNL measure to the user input for the requests of disambiguation type (2)2 for a second iteration

Figure 4. The effectiveness of the NL2DL in the domain of soccer against a keyword-based search

The experiments also tested if the natural language expression’s grammatical structures where successfully mapped to ontological structures (figure 2) and if the semantic relatedness measure resulted in satisfactory matches (figure 3) for the domain of soccer in a question answering system for the DS-MIRF Metadata Repository. We also presented the overall satisfaction of users with respect to the effectiveness of the results compared against a keyword-based search (figure 4). The conclusion that derived is that in a second iteration of tests the users expressed a higher satisfaction because their familiarity of using the system increased. The results that concerned ontological structures and semantics (figures 2 and 3) were strongly dependent on the form of the specific ontology. Overall, the performance decreases a little as the complexity of the natural language expression increases, but as shown in figure 4, we get the correct results sooner and faster against a keyword-based search.

6. Conclusions

We have presented the methodology of the automatic calculation of the OntoNL Semantic Relatedness measure for OWL ontologies. The motivation of this work came from the absence of a general, domain-independent semantic relatedness measure. The measure was successfully used for

2 One of the subject or object part of the language model cannot be disambiguated by using the ontology repository

natural language disambiguation and semantic ranking in the OntoNL Framework (Karanastasi 2007c). For the OntoNL Semantic Relatedness Measure evaluation, the framework takes into account a number of parameters regarding the characteristics of the ontologies involved and the types of users. We have focused our attention to the performance experimentation in a generic way utilizing readily available ontologies in the web, not carefully constructed by hand, ontologies.We concluded to the parameters that affect the choice of the weight value for each one of the sub-measures developed to comprise the OntoNL measure and we used the evaluation empirical results and Linear Programming to define the values of these weights by defining ontology metrics that influence the weights of the OntoNL measure. The methodology showed that with the correct definition of ontology metrics we get realistic results for the relatedness of concepts of a domain ontology. The methodology was based on the feedback of the users we used for the experimentation. By using a more systematic way of extracting the knowledge and experience of the users we may get a more accurate definition of ontology metrics with even better results in comparison with human judgments.Finally, we have presented an implemented application environment for the OntoNL Framework that addresses a semantic multimedia repository with digital audiovisual content of soccer events and metadata concerning soccer in general. This application was developed and demonstrated in the 2nd and 3rd Annual Review of the DELOS II EU Network of Excellence (IST 507618) (http://www.delos.info/ ). We have evaluated the OntoNL Semantic Relatedness Measure’s performance using this application environment.

References

N. Guarino, “Formal Ontology and Information Systems”, in Proc.of FOIS ’98, Trento, Italy, 6-8 June, 1998.A. Karanastasi, A. Zwtos, and S. Christodoulakis, “User Interactions with Multimedia Repositories using Natural Language Interfaces - OntoNL: an Architectural Framework and its Implementation”, in Journal of Digital Information Management - JDIM, Volume 4, Issue 4, December 2006C. Fellbaum. WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, 1998.P. Resnik. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, 1995.R. Rada and E. Bicknell. Ranking documents with a thesaurus. JASIS, 40(5):304–310, 1998.J. J. Jiang and D. W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics, 1997.A. Budanitsky, and G. Hirst. Evaluating WordNet-based Measures of Lexical Semantic Relatedness. Computational Linguistics 32(1): 13-47M. Jarmasz and S. Szpakowicz. Roget’s thesaurus and semantic similarity. In RANLP04 Conference Proceedings, pages 212–129, 2003.A. Karanastasi and S. Christodoulakis, “Semantic Processing of Natural Language Queries in the OntoNL Framework”, in the Proceedings of the IEEE International Conference on Semantic Computing (IEEE ICSC), 17-19 September 2007, Irvine, CA, 2007aS. Tartir, I.B. Arpinar, M. Moure, A. P. Sheth, and B. Aleman-Meza, “OntoQA: Metric-Based Ontology Quality Analysis”,

Journal of Digital Information Management q Volume 6 Number 5 q October 2008374

in Proc. Of IEEE Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources, 2005D. Vrandecic and Y. Sure. “How to design better ontology metrics”, in Proc. 4th European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, 2007B. Gärtner, and J. Matousek, “Understanding and Using Linear Programming”, Berlin: Springer, 2006C. Tsinaraki, S. Christodoulakis. “ A Multimedia User Preference Model that supports Semantics and its application to MPEG 7/21”, In Proceedings of MMM 2006, Beijing, China, 4-6 January 2006

A. Karanastasi, A. Zotos, S. Christodoulakis. “The OntoNL Framework for Natural Language Interface Generation and a Domain-Specific Application”. DELOS Conference 2007b: 228-237D. Radev, H. Qi, H. Wu, W. Fan. “Evaluating Web-based Question Ansering Systems”, Proceedings of LREC, 2002C. Tsinaraki, P. Polydoros, S. Christodoulakis, “Interoperability support for Ontology-based Video Retrieval Applications”, in the Proceedings of CIVR 2004, Dublin/Ireland, July 2004A. Karanastasi, and S. Christodoulakis, “Ontology-Driven Semantic Ranking for Natural Language Disambiguation in the OntoNL Framework”, in Proc. 4th European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, 2007c