University of Sheffield NLP
Outline
• NLIs to ontologies and their usability
• QuestIO – Question-based Interface to Ontologies
• Towards better usability using FREyA• Demo and evaluation
• Conclusion
210/12/09 Danica Damljanović
University of Sheffield NLP
Motivation
Large datasets such as Linked Open Data available
SPARQL/SeRQL: complex syntax: not easy to learn writing queries is error-prone task requires understanding of Semantic Web
technologies
University of Sheffield NLP
Objective
Allow domain experts to query knowledge in RDF/OWL format in a user-friendly manner
University of Sheffield NLP
Danica Damljanović
Natural Language user-friendly?
(Kaufmann and Bernstein, 2007)• Natural Language Interfaces preferred to keywords, menu-
guided, and graphical interfaces
(Linckels, 2007): • keywords preferred to NL interfaces
University of Sheffield NLP
6Danica Damljanović
NLIs to ontologies
University of Sheffield NLP
Error rate caused by…
• Users not familiar with the covered knowledge
• Knowledge is not available, but the system is not making that clear to the user i.e. feedback messages not helpful
• Users have assumptions/misconceptions about the system capabilities and supported language
University of Sheffield NLP
8Danica Damljanović
Usable NLIs to KBs: challenges
• Robustness
• Portability
• What to show?
• Understanding information need
• Habitability
University of Sheffield NLP
Challenging habitability problem
University of Sheffield NLP
Question-based Interface to Ontologies
• Robust• Zero customisation!• Easy to use: no training for
the user.• Deal with incorrectly
formulated queries• Accept queries of any
length and form.• Automatically ranks results
and shows the highest rank to the user
University of Sheffield NLP
NL --> SeRQL query
Filtering concepts
Ranking concepts
Query Creator
Query Execution
University of Sheffield NLP
An Example
1.15
1.19
compare
University of Sheffield NLP
FREyA (Feedback, Refinement, Extended Vocabulary Aggregator)
• assist the user formulate the query and express his need more precisely
• Implement user-system interaction and learn from it
University of Sheffield NLP
University of Sheffield NLP
Example 1
geo:City
geo:State new york
POC
POC
population
geo:cityPopulation
University of Sheffield NLP
New York is a city
University of Sheffield NLP
New York is a state
University of Sheffield NLP
POC
POC
POC
state
area geo:stateArea geo:State
geo:isLowestPointOf
lowest point
Example 2
University of Sheffield NLP
Demo
• http://gate.ac.uk/freya
University of Sheffield NLP
In lab evaluation• Mooney dataset: geography of the United States
• Relatively small ontology
• 250 questions
• require a high level of understanding of semantic meaning
• Recall and precision:
• 76% when we identify only the answer type correctly (first POC in the question)
• Expected: ~91% when we include all POCs from the question and add more refined suggestions (such as max or min for datatype properties of type number)
University of Sheffield NLP
Next steps
• Improvement of the learning mechanism
• User-based evaluation
• Trying it with some other datasets
University of Sheffield NLP
Thank you!
• Questions?