Understanding Naturally Conveyed Explanations of

Device Behavior

Michael Oltmans and Randall Davis

MIT Artificial Intelligence Lab

Michael Oltmans

Roadmap

The problem• Our approach• Implementation

– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding

• Evaluation and contributions• Related and future work

Michael Oltmans

Sketches Models

• We have a sketch of a device

• A simulation model can be generated from the sketch

• Life is good… or is it?

Michael Oltmans

Michael Oltmans

The Problem

• No representation of intended behavior

• People talk and sketch but the computer doesn’t understand

Michael Oltmans

Task

• Understand descriptions of device behavior:– Given:

• A model of the device’s structure• A natural explanation of the behavior

– Generate a causal model of behavior

Michael Oltmans

Michael Oltmans

Roadmap

• The problemOur approach• Implementation

– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding

• Evaluation and contributions• Related and future work

Michael Oltmans

Naturally Conveyed Explanations

• Natural input modalities– Sketched devices– Sketched gestures– Speech

• Natural content of descriptions– Causal– Behavioral

Michael Oltmans

Example: Describing the Behavior of a Spring

Tool: Description:

Michael Oltmans

Example: Describing the Behavior of a Spring

Tool: Description:

Mechanical CAD

Spring length = 2.3cm Rest length = 3.0cm

Michael Oltmans

Example: Describing the Behavior of a Spring

Tool: Description:

Mechanical CAD

Spring length = 2.3cm Rest length = 3.0cm

Qualitative Reasoner

(< (length spring) (rest-length spring))

Michael Oltmans

Example: Describing the Behavior of a Spring

Tool: Description:

Mechanical CAD

Spring length = 2.3cm Rest length = 3.0cm

Qualitative Reasoner

(< (length spring) (rest-length spring))

ASSISTANCE“The spring pushes the block”

Michael Oltmans

Sources of power

• Conventions in explanations aide interpretation– Description order suggests causal order

• Constrained vocabulary

• Overlapping descriptions provide constraints on interpretations

Michael Oltmans

Roadmap

• The problem• Our approach Implementation

System architectureHow ASSISTANCE interprets descriptionsDemonstrating understanding

• Evaluation and contributions• Related and future work

Michael Oltmans

ASSIST

•Recognize sketch

ASSIST

•Recognize sketch

Sketch Speech

Causal Model

and Simulation

Causal Model

and Simulation

ViaVoice™•Recognize speech•Parse

ViaVoice™•Recognize speech•Parse

LTRE•Truth Maintenance•Rule System

LTRE•Truth Maintenance•Rule System

ASSISTANCE

•Interpret explanation

Michael Oltmans

Outputs

• Consistent causal model– Tree– Nodes are events– Links indicate causal relationships

• Demonstration of understanding– Natural language descriptions of causality– Parameter constraints

Michael Oltmans

The Representation of Utterances

• Input comes from ViaVoice™ :– Grammar constructed based on observed

explanations– Tagged with parts of speech and semantic

categories

Michael Oltmans

Representing the parse tree

SENTENCE SIMPLE_SENTENCE

(… “body 1 pushes body 2” (S0) t1)

SENTENCE SIMPLE_SENTENCE

(… “body 1 pushes body 2” (S0) t1)

DIRECT_OBJECT NOUN NOUN-PHRASE

(… “body 2” (S0 t1 t3) t5)

DIRECT_OBJECT NOUN NOUN-PHRASE

(… “body 2” (S0 t1 t3) t5)PROPELS VERB

(… “pushes” (S0 t1 t3) t4)

PROPELS VERB

(… “pushes” (S0 t1 t3) t4)

SUBJECT NOUN NOUN-PHRASE

(… “body 1” (S0 t1) t2)

SUBJECT NOUN NOUN-PHRASE

(… “body 1” (S0 t1) t2)VERB_PHRASE

(… “pushes body 2” (S0 t1) t3)

VERB_PHRASE

(… “pushes body 2” (S0 t1) t3)

“body 1 pushes body 2”

Michael Oltmans

Steps In Interpreting Explanations:

• Infer motions from annotations and build event representations

• Find causal connections

• Search for consistent causal structures

• Pick best causal structure

Michael Oltmans

Step 1: Inferring Motions from Annotations

• Inputs:– Arrows– Utterances

• “moves,” “pushes,” “the spring releases”

• Outputs:– (moves body-1 moves-body-1-394)– (describes arrow-2 moves-body-1-394)

Michael Oltmans

•Rule triggers:–Arrow–Arrow referent (i.e. a body)–The body is mobile

•Rule body records that:–The body moves–The arrow describes the path

Inferring Motion From Arrows

Michael Oltmans

(rlet ((?id (new-id “Moves” ?name))) (rassert! (:implies (:AND ?f1 ?f2 ?f3) (:AND (moves ?body ?id) (describes ?arrow ?id)))

:ARROW-IS-MOTION)))

Inferring Motion From Arrows(rule ((:TRUE (arrow ?arrow) :VAR ?f1) (:TRUE (arrow-referent ?arrow ?body)

:VAR ?f2) (:TRUE (can-move ?body) :VAR ?f3) (:TRUE (name ?name ?body)))

Michael Oltmans

Multi-Modal References

• Match a sentence whose subject is “this” and a pointing gesture

• Assert the referent as the subject of the sentence

• Limitations:– User must point at referent before the

utterance– Allow one “this” per utterance

Michael Oltmans

Redundant Events

• Redundant explanations lead to multiple move statements for some events

• Merge them into a unique event statement

“Body 1” falls

(moves body-1 id-1)

(moves body-1 id-2)

Event 1

Michael Oltmans

Step 2: Find Causal Connections

• Plausible causes– Arrow indicating motion near another

object– Exogenous forces

• Definite causes– “When … then …” utterances– “Body 1 pushes body 2”

Michael Oltmans

Step 3: Search for Consistent Causal Structures

• Some events have several possible causes

• Find consistent causal chains

• Search– Forward looking depth-first-search– Avoids repeating bad choices by recording

bad combinations of assumptions

Michael Oltmans

Step 4: Find the Best Interpretation

• Filter out interpretations that have unnecessary exogenous causes

• Pick the interpretation that most closely matches the explanation order

• While there are multiple valid interpretations– Choose one event with multiple possible causes– Assume the causal relation whose cause has the

earliest description time

Michael Oltmans

Answer Queries and Adjust Parameters

• Queries: – Designer: What is body 2 involved in?– ASSISTANCE: The motion of body 3 causes

the motion of body 2 which causes the motion of body 5

• Parameter Adjustment– Set spring length

Michael Oltmans

Roadmap

• The problem• Our approach• Implementation

– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding

Evaluation and contributions• Related and future work

Michael Oltmans

Limitations of the Implementation

• Scope of applicability restricted– State transitions are one step deep– Cannot handle conjunctions of causes

• Limited knowledge about common device patterns– Latches, linkages, etc…– Supports and prevents

• Natural language limitations– Use a full featured NL system like START– Formally determine the grammar

Michael Oltmans

Evaluation of the Approach

• Advantages– Focus on behavior in accordance with survey

results– Move away from rigidity of WIMP interfaces– Similar to person-to-person interaction

• Alternatives– More dialog and feedback– Natural vs. efficient– Open claim that the domain is adequately

constrained

Michael Oltmans

Contributions

• Understanding naturally conveyed descriptions of behavior

• Generating representations of device behavior– Match the designer’s explanation– Generate simple explanations of causality– Allow the calculation of simulation

parameters

Michael Oltmans

Related Work• Understanding device sketches

– Alvarado 2000

• Multimodal interfaces– Oviatt and Cohen

• Causality– C. Rieger and M. Grinberg 1977

Michael Oltmans

Future Work

• Direct manipulation

• Dialog

• Expand natural language capabilities

• Smart design tools