Mental Models for Human-Robot Interaction

Christian Lebiere (cl@cmu.edu)1

Florian Jentsch and Scott Ososky2

1Psychology Department, Carnegie Mellon University2Institute for Simulation and Training, University of

Central Florida

Cognitive Models of Mental Models

• Mental models provide a representation of situation, various entities, capabilities, & past decisions/actions

• Current models are non-computational descriptions• Cognitive models can provide computational link to

overall robotic intelligence architecture for dual uses:– Provide a quantitative, predictive understanding of human

team shared mental models– Support improved design of human-robot interaction tools

and protocols– Provide a cognitively-based computational basis for

implementation of mental models in robots

Representation Components

• Mental model representation– Ontology of concepts and decisions

• Lexical (WordNet), Structural (FrameNet), Statistical (LSA)

– Symbolic frameworks• Decision trees, semantic networks

– Statistical frameworks• Bayesian networks, semantic similarities

• Knowledge of task situation– Situation awareness – mapping to levels of SA– Environment limitations – who sees/knows what (perspective)– Architectural limitations – who remembers what (WM, decay)

Reasoning and inference

• Inferring mental models– Instance-based learning (Gonzalez & Lebiere)

• E.g., Learning to control systems by observation or imitation

• Inferring current knowledge– Perspective-taking in spatial domain (Trafton)

• E.g., hide and seek, collaborative work

• Predicting decisions– Theory of mind recursion (Trafton, Bringsjord)– Imagery-based simulation (Wintermutte)– Shared plan execution in MOUT (Best & Lebiere)– Sequence learning in game environments (West & Lebiere)

Cognitive Architectures• Computational representation of

invariant cognitive mechanisms• Behavior selection

– Production systems– Utility – rewards and costs

• Memories– Working memory: buffers– Long-term: semantic/episodic– Activation mechanisms

• Learning– Symbolic and statistical

• Human factor limitations– Perceptual-motor parameters

• Individual differences– Strategies and knowledge– Capacity parameters

EnvironmentPr

oduc

tions

(Bas

al G

angl

ia)

Retrieval Buffer (VLPFC)

Matching (Striatum)

Selection (Pallidum)

Execution (Thalamus)

Goal Buffer (DLPFC)

Visual Buffer (Parietal)

Manual Buffer (Motor)

Manual Module (Motor/Cerebellum)

Visual Module (Occipital/etc)

Intentional Module (aPFC)

Declarative Module (Temporal/Hippocampus)

Pursuit Task

• Follow that Guy: human soldier and robot teammate– Shared mental model of pursuit situation scenario

• Set of data encoding various scenarios• Items organized according to SMMs held by expert teams

(Equipment, Task, Team Interaction, Team)• Decision tree built using information from police “foot

pursuit” procedures• For each decision, the most critical item is listed

– However, other factors may be considered in weighing decision

• Loop to end or continue the pursuit given fluid situation

Data

Scenario Data and Decision Tree

Part 1: Who should pursue?

Start

H-R Communication reliable (5x5)?

Is the terrain negotiable for robot?

Are suspects armed?

Robot only pursuit

Soldier only pursuit Team pursuitHold position,

report incident

Continue to Part 2: pursuit loop

Is the threat immediate (civilians, etc.)

Are sensors reliable in the search area?

Current last known location?

YES YES YES YES

YES

YES

No No No No

Is backup support available?

Immediate threat / critical situation?

No

NoNo

YES

No

YES

EQ-C3 SK-E3 EQ-S3 SK-S2 SK-S8

IA-A1 SK-S8 SK-S7

Is the suspect armed?

Was this, or is there potential for a violent crime?

Can a perimeter be set up to contain the suspect?

Do you have supervisor clearance?

Deciding whether to

pursue

Do you know the identity of the suspect?

Are backup units available to assist you?

Begin or Continue pursuit Do you have line

of sight with suspect?

Can you apprehend them at a later time?

What are the traveling surface conditions?

What is the pedestrian traffic like?

What are the weather conditions?

YesNo NoYes YesNo No Yes

Yes No Yes No Yes No

Are communications functioning properly?

Yes No

Light/ ModerateHeavy Good/

Fair PoorGood/

Fair PoorYes No

SK-S2SK-S1 SK-S3 SK-A1

IA-A1 EQ-C3 SK-A2

IA-R1

TM-W1 SK-E1 SK-E2 SK-E3

General Cognitive Model• Develop general model that takes mental models in the form

of decision trees and learns to retrieve and execute them• Each decision is represented as sequence of chained steps• Each piece of data is represented as separate chunk• Model (7 p* production rules) depends on declarative memory

to retrieve rule steps, data items and decision instances– No hardcoded decision logic

• Each decision depends on matching against past instances combining activation recency, frequency and partial matching

• Stochasticity of activation results in probabilistic decisions• Run model in Monte Carlo mode for decision distribution• Cross-validation: train on some scenarios, test on others

Individual Decision Inference

Overall Decision Agreement

Generalized Condition

• 35 scenarios• 3 experts• Intermediate

decisions• Relative

rankings• Desirability

ratings• Comments

Results• Match to first-last

ranks, poor middle• Slightly different

ratings pattern• Comparable cross-

expert correlations

Learning

• Proceduralize individual steps from declarative instructions to production rules to replicate learning curve from novice to proficiency and expertise

• Apply feature selection using utility learning to encode and use only a subset of data items for each decision

• Learn shortcuts that combine multiple individual binary decisions into single, multi-outcome decision

• Generate rankings/ratings from probability judgments generated from activation of memory retrievals

• Abstract decision instances into discrete types

Future Work

• Validate model against human participants data along entire learning curve and broad range of situations

• Explore Bayesian network formalism as alternative to enhance generalization in multi-step decisions

• Integrate cognitive model in multi-agent simulations to validate computational mental model in dynamic decision-making setting

• Integrate computational cognitive model on robotic platform to assess ability to improve human-robot interaction through shared models