Post on 01-Apr-2015
transcript
Usability Enhancement Methods in Natural Language Interfaces for
Querying OntologiesBirmingham, 12 April, 2011
Danica DamljanovićUniversity of Sheffielddanica@dcs.shef.ac.uk
OutlineBackground:
What are Ontologies?What are Natural Language Interfaces (NLIs)?What are Usability Enhancement Methods?
ObjectiveImprove NLIs to Ontologies with usability enhancement
methodsOur approach
Two NLI systems for querying ontologies: QuestIO FREyA
Two usability studies to test the usability enhancement methodsFindingsDemo
Conclusion
Mary works for University of Sheffield, which is located in Sheffield. Sheffield is located in the United Kingdom. Mary lives in Sheffield.
MARY <is a> PERSONUNIVERSITY OF SHEFFIELD <is an> ORGANISATIONMARY <works for> UNIVERSITY OF SHEFFIELDSHEFFIELD <is a> CITYUNIVERSITY OF SHEFFIELD <is located in> SHEFFIELDUNITED KINGDOM <is a> COUNTRYSHEFFIELD <is located in> UNITED KINGDOMMARY <lives in> SHEFFIELD
SELECT ?countryWHERE{?person <lives in> ?city?city <located in> ?countryFILTER ?person = MARY}
4
In which country does Mary live?
What are Usability Enhancement Methods?Who are the users?
application developers
end users
The Objective
Increase usability of Natural Language Interfaces to ontologies For end users: increase precision and recall For application developers: decrease the time
for customisation
Our ApproachScope Ranking Lexicon Usabilit
y methods
Grammar analysis
Supported language
QuestIO
one ontology
ontology structure, string similarity
ontology lexicalisaions
none (automatic)
Shallow (morphological analysis)
grammatically correct/question fragments
FREyA a set of ontologies/repository
string similarity, synonym detection, user
ontology lexicalisations, Wordnet, user vocabulary
feedback, clarification dialogs (user interaction)
Deeper (parsing)
grammatically correct/question fragments
QuestIO
1.15
1.19
compare
QuestIO prototype
QuestIO: User EvaluationUsability testing:
effectiveness: could the tasks could be finished using QuestIOefficiency: how quickly?user satisfaction
System Usability Scale (SUS) subjective (was it easy to formulate a query?, etc.)
Experimental setup: a complete counterbalanced repeated measures, task-based
evaluation design Baseline (search engines) vs. QuestIO
12 subjects familiar with the domain (GATE software) four tasks:
three defined, e.g. ...find parameters of Cebuano gazetteer... one undefined task, ...find anything you want about GATE software...
QuestIO User Evaluation: ResultsEffectiveness:
the scale from 0 (easy) to 2 (impossible)
0.355 for QuestIO in comparison to 0.895 for baseline, p = 0 .001
Efficiency: the subjects significantly
slower when using baseline (157s) in comparison to QuestIO (107s), p=0.001
User satisfaction: SUS score satisfactory (69.38)
Tasks: defined tasks: user
satisfaction reaching 90%
undefined tasks: user satisfaction low (~44%)
QuestIO: weaknessesLexical failures: Tokenizer vs. TokeniserConceptual failures:
missing concepts, relations, or bothThe users not being aware of why the
failures happenedCan this be improved with usability
enhancement methods such as feedback and clarification dialogs?
15
FREyA - Feedback, Refinement, Extended Vocabulary Aggregator
Feedback: showing the user system interpretation of the queryRefinement:
resolving ambiguity: generating dialog whenever one term refers to more than one concept in the ontology (precision)
Extended Vocabulary: expressiveness: generating dialog whenever an “unknown” term
appears in the question (recall) portability: no need for customisation from application developers
The dialog: generated by combining the syntactic parsing and ontology-based
lookup the system learns from the user’s selections
Feedback: answer is found
Feedback: No answer is found
Feedback: User EvaluationUsability testing:
effectiveness efficiency user satisfaction
System Usability Scale (SUS) subjective (was it easy to formulate a query?, etc.)
Experimental setup: 30 subjects outside Sheffield, two domains (GATE software and
US geography) four tasks:
three defined: two repeated from the previous study one where the answer was not available, e.g. ...find states bordering
hawaii... one undefined task, ...find anything you want about GATE software or
rivers, cities, ... in the United States...
Does the feedback make any difference?Effectiveness: yes , p=0.01, 0.67 for QuestIO, 0.13 for
FREyAEfficiency: no, although the overall result differs (180.5
seconds for QuestIO, 155.27 seconds for FREyA), 2-tailed independent t-test reveals that this difference is not significant (p=0.852)
Query Formulation: for the defined tasks there is no difference in the perception of the difficulty of the supported language (F=5.255, p=0.071), but for the undefined tasks the users believed that the language supported by FREyA is easier! (F=8.016, p=0.015)
Showing that the system knows about certain concepts, but cannot find any relation between them was not clear.
Interactive features were well accepted.
FREyA Workflow
ESWC 2010 21
Demo
03 June 2010
http://gate.ac.uk/freya
22
Evaluation: correctness Mooney GeoQuery dataset, 250 questions
34 no dialog, 14 failed to be answered
Precision=recall=94.4%
23
Evaluation: Learning 10-fold cross-
validation
202 Mooney GeoQuery questions that could be correctly mapped into SPARQL and required dialog
improvement from 0.25 to 0.48
Errors: ambiguity and sparseness
Evaluation: Ranking
Mean Reciprocal Rank: 0.76
Learning the Correct Ranking Randomly selected 103 dialogs from 202 questions (343
dialogs)
MRR increased for 6% from 0.72 to 0.78
26
Evaluation: Answer Type
45.60%
53.20%
0.01%
Answer TypeCorrect (1 dia-log)
27
ConclusionCombining syntactic parsing with ontology-
based lookup in an interactive process of feedback and query refinement can increase the precision and recall of NLIs to ontologies,
while reducing the time for customisation by shifting some tasks from application developers to end users.
More information... D. Damljanovic, M. Agatonovic, H. Cunningham: FREyA: an Interactive Way of
Querying Linked Data, 1st Workshop on Question-Answering over Linked Data, in conjunction with ESWC’11, 2011. (to appear)
D. Damljanovic, M. Agatonovic, H. Cunningham: Natural Language Interfaces to Ontologies: Combining Syntactic Analysis and Ontology-based Lookup through the User Interaction. In Proceedings of the 7th Extended Semantic Web Conference (ESWC 2010), Springer Verlag, Heraklion, Greece, May 31-June 3, 2010. PDF
D. Damljanovic, M. Agatonovic, H. Cunningham: Identification of the Question Focus: Combining Syntactic Analysis and Ontology-based Lookup through the User Interaction. In Proceedings of the 7th Language Resources and Evaluation Conference (LREC 2010), ELRA 2010, La Valletta, Malta, May 17-23, 2010. PDF
D. Damljanovic. Towards portable controlled natural languages for querying ontologies. In Rosner, M., Fuchs, N., eds.: Proceedings of the 2nd Workshop on Controlled Natural Language. Lecture Notes in Computer Science. Springer Berlin/Heidelberg, Marettimo Island, Sicily (September 2010)