02/04/09 Danica Damljanović 1
Natural Language Interfaces to conceptual models: usability and performance
Danica Damljanović
University of Sheffield NLP
Outline
• NLIs to KBs and their usability
• QuestIO – Question-based Interface to Ontologies Demo and evaluation
• Towards better usability using FREYA
• Conclusion
202/04/09 Danica Damljanović
University of Sheffield NLP
Natural Language Interfaces
• Allow users to interact with a system using written or spoken language to perform tasks which require knowledge of a formal query language NLIs to structured data:
• NLIs to DBs: TEAM, PRECISE• NLIs to KBs
NLIs to semi-structured data• Open-domain question answering systems
Other:• E.g. NLC: replacement for a programming language• Dialog and tutoring systems
University of Sheffield NLP
Danica Damljanović
NLIs to knowledge bases
(Kaufmann and Bernstein, 2007)
• Natural Language Interfaces preferred to keywords, menu-guided, and graphical interfaces
(Linckels, 2007): • keywords preferred
to NL interfaces
University of Sheffield NLP
8Danica Damljanović
NLIs to KBs
University of Sheffield NLP
9Danica Damljanović
Usability of NLIs
• Who uses NLIs? Application developers: customisation End users: search
• Usability Effectiveness Efficiency User satisfaction
University of Sheffield NLP
10Danica Damljanović
Usable NLIs to KBs: challenges
• Robustness
• Portability
• What to show?
• Understanding information need
• Habitability
University of Sheffield NLP
12Danica Damljanović
Habitability
• Can the User and the System speakthe same language?
University of Sheffield NLP
14Danica Damljanović
Question-based Interface to Ontologies
University of Sheffield NLP
16Danica Damljanović
NL --> SPARQL query
Filtering concepts
Ranking concepts
Query Creator
Query Execution
University of Sheffield NLP
An Example
1.15
1.19
compare
University of Sheffield NLP
Demo
• http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/questio/questio.html
• http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/prototype-tutorial/prototype-tutorial.html
20Danica Damljanović
University of Sheffield NLP
User-centric task-based evaluation
• Training: using video tutorials
• 12 participants, 4 tasks: 3 defined, e.g. “Find runtime parameters of
Cebuano gazetteer.” 1 free task: “Think of any task that you would
like to perform using this prototype.”
• Measured: Efficiency, Effectiveness and User satisfaction
26Danica Damljanović
University of Sheffield NLP
Efficiency: average time per task
University of Sheffield NLP
Effectiveness: how successfully the tasks were finished?
University of Sheffield NLP Post –task survey: question 1
University of Sheffield NLP
Post –task survey: question 2
University of Sheffield NLP
Post –task survey: question 3
University of Sheffield NLP
Post –task survey: question 4
University of Sheffield NLP
Evaluation: conclusion• Browsing ontology mostly helpful – but
only for users who are familiar with ontologies
• Refinement pane: primitive, still favourable for defined tasks
• Tasks not finished: in cases when the user information need was
not precisely expressed (undefined tasks) When the answer was not in the ontology
34Danica Damljanović
University of Sheffield NLP “find out which are the runtime parameters of Cebuano Gazetteer”
• “cebuano gazetter parameters”• “ What are the runtime parameters of cebuano
gazetteer?“• “what are the parameters of cebuano gazetteer?”• “Cebuano gazetteer runtime parameters“• “Runtime parameters of cebuano gazetteer“• “Cebuano runtime parameters“• “Cebuano gazeteer“>>“Cebuano gazetteer“
Find parameters by browsing the ontology35Danica Damljanović
University of Sheffield NLP
“runtime parameters of Cebuano Gazetteer”
• “Cebuano runtime parameters“GATE plugin Resource Parameter
Processing Resource
Contains
Resource hasInitTimeParameter
hasRuntimeParameter
University of Sheffield NLP
Undefined tasks
• “Developer of Tokeniser”• ”Projects about GATE ”, ”GATE web site”• ” Tokenizer ” vs. ” Tokeniser ”• ” Gazetteer ” vs. ”Gazeteer ”• ”horacio saggion publications” vs. ”horacio
saggion articles”• ”Author of morphological analyser ” >>
”Developer of morphological analyser ” >> ”Developer and morphological analyser ”
3704/06/09 Danica Damljanović
University of Sheffield NLP
Next steps: FREyA
38Danica Damljanović
FREyA (Feedback, Refinement, Extended vocabulary, Agregation)
University of Sheffield NLP
Example
• the user's query: Cities in Europe
• FREyA: Cities >>locatedIn >>Countries >> locatedIn >>
Europe Cities >>locatedIn >> Countries >> partOf >> Europe
• Attempto: which cities are located in countries that are
part of Europe?
3904/06/09 Danica Damljanović
University of Sheffield NLP
Challenges
• Return answers in real time: which rivers flow through Germany >>
which rivers flow through (cities in) Germany
• Refinement model - an ontology? with concepts such as query (hasAnswer)
answer, refinement, feedback, subject of query (main subject), domain, context.
find out what the user wants and instantiate the refinement ontology.
40Danica Damljanović
University of Sheffield NLP
41Danica Damljanović
Expected contribution
• Usability improvement of NLIs: Eliminate training Improve the performance by implementing
user-system interaction using FREYA:• help the user to easily familiarize himself
with the system capabilities• express his need more precisely, in a way
which is understandable by the system
University of Sheffield NLP
42Danica Damljanović
Thank you
• Questions?
University of Sheffield NLP
References• (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M.
Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001.
• (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004.
• (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM.
• (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006.
• (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007.
• (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.