Daniel Sonntag | 1

LREC 2010Speech Grammars for Textual Entailment Patterns in Multimodal QuestionAnswering

Daniel Sonntag, Bogdan Sacaleanu, DFKI21/05/2010

Daniel Sonntag | 2 | 2

Outline

» Semantic Dialogue Shell

» Textual Entailment

» Processing Example

» Conclusions

Daniel Sonntag | 3

Acknowledgements

»Thanks go out to Robert Nesselrath, Yajing Zang, Günter Neumann, Matthieu Deru, Simon Bergweiler, Gerhard Sonnenberg, Norbert Reithinger, Gerd Herzog, Alassane Ndiaye, Tilman Becker, Norbert Pfleger, Alexander Pfalzgraf, Jan Schehl, Jochen Steigner, and Colette Weihrauch for the implementation and evaluation of the dialogue infrastructure.

RobustQuestion

Understanding

Ease the interface to external third-party

components.

SPARQL

Daniel Sonntag | 4 | 4

Semantic Dialogue Shell

Daniel Sonntag | 5

Dialogue Shell Workflow

SpeechInterpretation

TextInterpretation

GestureInterpretation

GraphicGeneration

TextGeneration

SpeechInterpretation

ModalityFusion

PresentationPlanning

Pe

rso

na

lis

ati

on Dialogue

and

Interaction

Management

InteractiveSemanticMediator

InteractiveServiceCompo-sition

eTFS/SPARQL

SPARQL

SPARQL

SPARQL

OWL-API

Visualisation VisualisationService

ExternalInformation

Sources

Semantiic (Meta)Services

RDFKOIOS(Yago

Ontology)

RemoteLinked Data

Services

OWLAOIDE(Music

Ontology)

TextSummarisation

- Domain Model- Context Model- User Model

Daniel Sonntag | 6

THESEUS’s Semantic Dialogue Shell: Goals and Requirements

»Multimodal interaction with the Semantic Web and the Internet of Services

»Components customisable to different use case scenarios

»Flexible adaptation to» Input and output modalities» Interaction devices» Knowledge bases

»To understand a greater number ofqueries:

» Robust question understanding (NLU) when using both speech and written text input

» Semantic (i.e., a RDF or OWL based) query interpretation » The combination of robust question understanding and

ontology-based answer retrieval

Daniel Sonntag | 7

SPARQL Query Editor

SPARQL

Daniel Sonntag | 8

Daniel Sonntag | 9

Speech Grammar

<utterance name="SHOW_CV_OF_PERSON"> <phrases> <phrase>zeige ?mir den [werdegang lebenslauf] [von zu] PERSON</phrase> <phrase>sage ?mir mehr über den [werdegang lebenslauf] von PERSON</phrase> <phrase>wie ist der [werdegang lebenslauf] von PERSON</phrase> </phrases> <semantic-interpretation> <object type="odp#TaskRequest"> <slot name="odp#fusion-confidence"> <value type="Float">1.0</value> </slot> <slot name="odp#hasContent"> <object type="dialogshell#BackendRetrievalTask"> <slot name="dialogmanager#backendComponent"> <value type="String">mediator:summarizer</value> </slot> <slot name="odp#hasContent"> <variable name="PERSON"/> </slot> </object> </slot> </object> </semantic-interpretation> </utterance>

Daniel Sonntag | 10 | 10

Textual Entailment

Our idea is that an NLU grammar for speech input can be reused to build more robust multimodal text-

based question understanding by automatically generating textual entailment patterns.

Daniel Sonntag | 11

Textual Entailment & Information Access

Request

Method1

Method2

Information (RDF)

Implicit Mapping

Method1: Speech / Semantic Grammars

• RDF/OWL reasoning

Method2: RTE• textual reasoning

Ontology (RDF/OWL)

Conceptual

Textual

Reasoning

Daniel Sonntag | 12

Textual Entailment through Alignments

» For textual entailment to hold we need:» text AND background knowledge hypothesis» but background knowledge should not entail hypothesis alone

»Background Knowledge» Unsupervised acquisition of linguistic and world knowledge from

general corpora and web» Acquiring larger entailment corpora» Manual resources and knowledge engineering

» Alignment-based TE and Background Knowledge» Preprocessing: POS, morphology, cognates» Representation: bag-of-words» Knowledge Sources: WordNet, Roget‘s Thesaurus, Wehrle

Thesaurus

Daniel Sonntag | 13

Argumentation

» Input modalities are usually interpreted according to separate models and aligned to a shared model (often coarse-grained).

»Present a method of interpretation based on a common model (propagated changes to multiple modalities).

»Built on the grammar for speech inputs = Leveraging Existing Speech GrammarKnowledge

Daniel Sonntag | 14 | 14

Processing Example

Daniel Sonntag | 15

Entailment Patterns and Possible Hypotheses

Daniel Sonntag | 16

Entailment Patterns and Alignment Engine

»Association-based word alignment. Three steps:

» lexical segmentation, when boundaries of lexical items are identified;

» correspondence, when possible similarities are suggested in line with some correspondence measures;

» alignment, when the most likely semantically similar word is chosen.

Daniel Sonntag | 17

Entailment Patterns and Alignment Techniques

»Question: What is the birthplace of Angela Merkel?

»Pattern: Where is Angela Merckel born?

»Filters on a full alignment.

Daniel Sonntag | 18

Entailment Patterns and Alignment Techniques

POS Filter: Exclude unlikely alignments based on POS. Allow for the additional mappings: verb to noun (i.e., born vs. birthplace)Lexical Semantic Resource Filter: WordNet (synonyms); Roget Thesaurus (conceptually related words)String Similarity Filter: Dice coefficient, Longest common subsequence ratio; submatches, misspellings

System of weights: nouns, verbs, and adjectives are better scored than function words.

Daniel Sonntag | 19

Dialogue Example

» (1) U: “Open my personal address book. What do you know about Claudia?”

» (2) S: “There’s an entry: Claudia Schwartz. The personal details are shown below. She lives in Berlin.” + Google Map Display of street coordinates.

» (3) U: “Which is Claudia’s favorite kind of music? Do you know the bands she likes most?”

» (4) S: “Nelly Furtado” + Displays videos obtained from YouTube. (Rest API)

» (5) U: “How did experts rate her last album?”» (6) S: Shows an expert review according to the BBC Linked Data Set.» (7) U: “Show me other news.”» (8) S: Opens a browser + Text field and a new agency Internet page

(featuring Angela Merkel)» (9) U writes: “Where was Angela Merkel born? / In which town was

Angela Merkel born?” etc.» (10) S: “She was born in Hamburg.”» (11) U speaks again: “And Barack Obama?”» (12) S: “He was born in Honolulu.”» (13) U: “Show me Angela Merkel’s career.”

Daniel Sonntag | 20

Touchscreen Installation

Daniel Sonntag | 21

Image Analysis in Biomedicine MEDICO

Retrieval and examination of 2D picture series

Daniel Sonntag | 22 | 22

Conclusions

Daniel Sonntag | 23 | 23

Conclusions

»We described a multimodal dialogue shell for QA and focussed on the robust multimodal question understanding task.

»The textual interpretation is based on automatically generated textual entailment patterns.

»As a result, we can deal with written text input and different surface forms more flexibly according to the derived entailment patterns.

Daniel Sonntag | 24

Method Comparison

» Method 1: Speech Grammars» Speech grammars are verbose» Requires full coverage of expected input» Hard-coded reasoning in rules» Example:

» Show me all pictures of X.» What pictures does X have?» Show me all images of X.

» Method 2: NLU Grammars» Use of Textual Entailment» NLU grammars are compact» Requires partial coverage of possible input» Example:

» Show me all pictures of X.» Entailed utterances:

» What pictures does X have?» Show me all images of X.

2

1