S.C. Shapiro

cse@buff

alo

Symbol Anchoring in aGrounded Layered Architecture

with Integrated Reasoning

Stuart C. Shapiro

Department of Computer Science and Engineeringand Center for Cognitive Science

University at Buffalo

shapiro@cse.buffalo.edu

S.C. Shapiro

cse@buff

alo

Based on

Stuart C. Shapiro & Haythem O. Ismail, “Anchoring in a grounded layered architecture with integrated reasoning,” Robotics and Autonomous Systems 43, 2-3 (May 2003) 97-108.

S.C. Shapiro

cse@buff

alo

Outline

• Introduction• Perceivable entities and properties• Attentional Structures• Actions• Time• Language• Examples• Summary

S.C. Shapiro

cse@buff

alo

Definition

“Anchoring is the problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world.”

[Silvia Coradeschi & Alessandro Saffiotti]

S.C. Shapiro

cse@buff

alo

Embodied Cassie

• A computational cognitive agent– Embodied in hardware– or Software-Simulated– Based on SNePS and GLAIR.

S.C. Shapiro

cse@buff

alo

Cassie, the FEVAHR(Foveal ExtraVehicular Activity Helper-Retriever)

S.C. Shapiro

cse@buff

alo

FEVAHR/Cassie in the Lab

S.C. Shapiro

cse@buff

alo

Crystal Cassie

First Person Perspective Views

S.C. Shapiro

cse@buff

alo

Patofil and Filopatfrom “The Trial, The Trail”A VR drama by Josephine Anstey et al.

S.C. Shapiro

cse@buff

alo

DeliveryAgent

Using Byron Weber Becker’s Java versionof Rich Pattis’ Karel the Robot

S.C. Shapiro

cse@buff

alo

Magellan ProTM Mobile Robotfrom

iRobot

S.C. Shapiro

cse@buff

alo

SNePS• Knowledge Representation and Reasoning

– Intensional Representation

– Propositions as Terms

• SNIP: SNePS Inference Package– Specialized connectives and quantifiers

• SNeBR: SNePS Belief Revision• SNeRE: SNePS Rational Engine (Acting Language)• Interface Languages

– SNePSUL: Lisp-Like

– SNePSLOG: Logic-Like

– GATN for Fragments of English.

S.C. Shapiro

cse@buff

alo

Interaction with CassieEnglish

(Statement, Question, Command)

(Current) Set of Beliefs[SNePS]

(Updated) Setof Beliefs[SNePS]

Actions[SNeRE]

(New Belief)[SNePS]

English sentence expressingnew belief answering question reporting actions

Answer[SNIP]

GATN Parser

GATN Generator

ReasoningClarification DialogueLooking in World

Reasoning

S.C. Shapiro

cse@buff

alo

Entities, Terms, Symbols, Objects

• Cassie’s mental entity: a person named Stu

• SNePS term: B4

• Object in world:

S.C. Shapiro

cse@buff

alo

GLAIR Architecture

Knowledge Level

Perceptuo-Motor Level

Sensory-Actuator Level NL

Vision

Sonar

MotionProprioception

Grounded Layered Architecture with Integrated Reasoning

SNePSMind

Body

World

IP Sockets

S.C. Shapiro

cse@buff

alo

Alignment

KL

Body (PML/SAL)

World

SNePS term

PML structure

Object/Phenomenon Action

S.C. Shapiro

cse@buff

alo

Outline

• Introduction

• Perceivable entities and properties

• Attentional Structures

• Actions

• Time

• Language

• Examples

• Summary

S.C. Shapiro

cse@buff

alo

World Objectsto Feature Tuples

<Height, Width, Texture, . . . >

WorldPML/SAL

S.C. Shapiro

cse@buff

alo

Feature Tuplesto KL Terms

<Height, Width, Texture, . . . >

PML/SALKL

ProperName(B4, Stu)

Alignment

S.C. Shapiro

cse@buff

alo

Incomplete PML-Descriptions

<Height, nil, nil, . . . >

PML/SALKL

Height(B4, B12)

S.C. Shapiro

cse@buff

alo

Unifying PML-Descriptions

<x, nil, nil, ... >

PML/SALKL

B20

B30

B31

B6

<nil, y, nil, ... >

<nil, nil, z, ... >

<x, y, z, ... >

Isa

Prop

Prop

S.C. Shapiro

cse@buff

alo

Outline

• Introduction• Perceivable entities and properties• Attentional Structures• Actions• Time• Language• Examples• Summary

S.C. Shapiro

cse@buff

alo

Deictic & Modality Registersfor being situated in the world

IYouNow

Vision...

PMLKLTerms denoting

CassieAddressee

Current timeCurrent state(s) of looking at x

.

..

S.C. Shapiro

cse@buff

alo

Outline

• Introduction• Perceivable entities and properties• Attentional Structures• Actions• Time• Language• Examples• Summary

S.C. Shapiro

cse@buff

alo

Primitive Actions Alignedwith PML/SAL Functions

<... >

PMLKL

M2(B4)

Find(B4)

SAL

Modality Registers

World

A KL symbol is a pivot coordinating different modalities.

S.C. Shapiro

cse@buff

alo

Outline

• Introduction• Perceivable entities and properties• Attentional Structures• Actions• Time• Language• Examples• Summary

S.C. Shapiro

cse@buff

alo

Aligning NOW using MTF

NOW

Semi-lattice of times, all of which contain *NOW,any of which could be meant by “now”Finite---only conceptualized times of conceptualized states

Maximal Temporal Frame based on *NOW

KL

PML

S.C. Shapiro

cse@buff

alo

Moving NOW with MTF

NOW

KL

PML

Moves when Cassie acts, newly observes a state,or is informed of a new state. Always includes times of states in modality registers.

S.C. Shapiro

cse@buff

alo

Providing a Feel for Time

NOW COUNT n

hom

0

KL

PML

t1 t2q

!before after

time duration!

S.C. Shapiro

cse@buff

alo

Outline

• Introduction• Perceivable entities and properties• Attentional Structures• Actions• Time• Language• Examples• Summary

S.C. Shapiro

cse@buff

alo

Aligning Lexemes/NL

“Stu”

PML/SALKL

ProperName(B4, Stu)

Isa(B4, person)

“person”

ctgynpr

ctgyn

numsing

Grammar NL

World

S.C. Shapiro

cse@buff

alo

Outline

• Introduction• Perceivable entities and properties• Attentional Structures• Actions• Time• Language• Examples• Summary

S.C. Shapiro

cse@buff

alo

Acting 1

S.C. Shapiro

cse@buff

alo

Acting 2

I found a red robot.I am looking at a red robot.

Follow a red robot.

S.C. Shapiro

cse@buff

alo

Acting 3

I went to a red robot.I am near a red robot.I am following a red robot.

I found a red robot.I am looking at a red robot.

Follow a red robot.

S.C. Shapiro

cse@buff

alo

Use of Indexicals 1

Come here.I found you.I am looking at you.

S.C. Shapiro

cse@buff

alo

Come here.

I came to you, Stu.I am near you.

Use of Indexicals 2

I found you.I am looking at you.

S.C. Shapiro

cse@buff

alo

Who am I?Your name is ‘Stu’and you are a person.

Who have you talked to?

I am talking to you.Talk to Bill.

I am talking to you, Bill.Come here.

Use of Indexicals 3

S.C. Shapiro

cse@buff

alo

Come here.

I found you.I am looking at you.

Use of Indexicals 4

S.C. Shapiro

cse@buff

alo

Come here.

I came to you.I am near you.

I found you.I am looking at you.

Use of Indexicals 5

S.C. Shapiro

cse@buff

alo

Who am I?

I talked to Stuand I am talking to you.

Your name is ‘Bill’and you are a person.

Who are you?I am the FEVAHRand my name is ‘Cassie’.

Who have you talked to?

Use of Indexicals 6

S.C. Shapiro

cse@buff

alo

Outline

• Introduction• Perceivable entities and properties• Attentional Structures• Actions• Time• Language• Examples• Summary

S.C. Shapiro

cse@buff

alo

Summary• KL terms denote mental entities.

• KL terms aligned to PML structures.

• PML variables/registers contain KL terms.

• PML variables/registers/structures grounded in world via sensors & effectors.

• PML attentional structures anchor symbols by changing only when motivated.

S.C. Shapiro

cse@buff

alo

Summary from Jackendoff

Ray Jackendoff, Foundations of Language,Oxford University Press, 2002, Fig. 11.1, p. 348.(Independent work)

S.C. Shapiro

cse@buff

alo

For More Information

• Personnel

• Manual

• Tutorial

• Bibliography

• ftp’able SNePS source code

• etc.

• http://www.cse.buffalo.edu/sneps/