1

Learning Prepositions for Spatial Relationships in BOLT

Soar Workshop 2012James Kirk, John Laird

6/21/2012

2

Outline

• Prepositions in BOLT• Compositional Spatial Relationships• Learning Process in Architecture with SVS• Projection• Conclusions

3

Prepositions/Spatial Relationships in BOLT

• “What is to the right of the pantry?”• “Put the triangle in the stove.”• “Pick up the red object to the right of the pantry.”

4

Spatial Relationships as Compositions

y-alignedz-alignedx-greater than

Z

Y

X

“right of”

“Intersecting”y-alignedz-alignedx-aligned

“Above”z-greater than

Other potential compositions:

“In front of”y-less thanz-alignedx-aligned

5

Learn new prepositions that map to spatial relations “The red triangle is right of the blue sphere”

6Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 1Perception

- Receive object information from

input link

- Create internal representations

Working Memory

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Human: “The red triangle is right-of the blue sphere” Word – Category

Mapping

7Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 2Message Parsing

- Parse the given sentence

- Create usable message

structures

“The red triangle is right-of the blue sphere”

Working Memory

LGSoar

Obj1- bl3- sp7

Obj2-re23- tr17

Obj3- bl3- sq8

relation-message ^word right-of ^o1 red, triangle ^o2 blue, sphere

Human: “The red triangle is right-of the blue sphere” Word – Category

Mapping

8Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 3Interaction

Stack

- Designate message purpose

as preposition learning

Working Memory

LGSoar

relation-message ^word right-of ^o1 red, triangle ^o2 blue, sphere

Interaction

Prep. LearningObj1- bl3- sp7

Obj2-re23tr17

Obj3- bl3- sq8

Human: “The red triangle is right-of the blue sphere” Word – Category

Mapping

9Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

Working Memory

Interaction

relation-message ^word right-of ^o1 red, triangle ^o2 blue, sphere

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learning

Prep Learning

right-ofred, triangle

blue, sphere

Human: “The red triangle is right-of the blue sphere” Word – Category

Mapping

10

Word – Category Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- Lookup the category of red

from smem

Working Memory

Noun Learning

red color

Prep Learning

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learningright-of

red, triangle

blue, sphere

Human: “The red triangle is right-of the blue sphere”

Indexing

11

Noun/Adjective – Perceptual Symbol

Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- Lookup the symbol for red + color from smem

Working Memory red color

Noun/Adjective – Perceptual Symbol Map

re23Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learningright-of

red, triangle

blue, sphere

re23, triangle

Human: “The red triangle is right-of the blue sphere”

Indexing

Word – Category Mapping

Word – Category Mapping

12

Noun/Adjective – Perceptual Symbol

Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- Update the descriptions with

perceptual symbols

Working Memory

Noun/Adjective – Perceptual Symbol Map

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learningright-of

red, triangle

blue, sphere

re23, trianglere23, tr17

bl3, spherebl3, sp7

Human: “The red triangle is right-of the blue sphere”

Indexing

Word – Category Mapping

13

Noun/Adjective – Perceptual Symbol

Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- Look for the objects matching the description

Working MemoryObj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learningright-of

re23, tr17

bl3, sp7

Obj2

Obj1

Human: “The red triangle is right-of the blue sphere”

Indexing

Word – Category Mapping

14

Obj2

Obj1

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- SVS updates model of objects in the world from the perception input.

Human: “The red triangle is right-of the blue sphere”

Working MemoryObj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learningright-of

Obj1Obj2Obj3Spatial Primitives

Prep Learning

Word – Category Mapping

15

Obj2

Obj1

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- SVS is queried for primitive

spatial relationships

between Obj1 and Obj2

Human: “The red triangle is right-of the blue sphere”

Prep Learning

Working MemoryObj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learningright-of

Obj1

Obj2

Obj3

Obj2

Obj1?

Spatial Primitives

Word – Category Mapping

16Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- The results of the query are

stored in working memory.

Human: “The red triangle is right-of the blue sphere”

Prep Learning

y -alignedz-aligned+x

Working Memoryright-of

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learning

Word – Category Mapping

17Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- List of spatial primitives are

abstracted into general spatial

relationship and mapped to “right-

of.”

Human: “The red triangle is right-of the blue sphere”

Prep Learning

Working MemoryObj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

SR11

y-aligned z-aligned +xright-of

Prep. Learning

Word – Category Mapping

18

Preposition – Spatial RelMapping

Preposition – Spatial RelMapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 4Preposition

Learning

- Learned spatial relationship mapping to “right-of” is

stored in semantic memory.

Human: “The red triangle is right-of the blue sphere”

Prep Learning

Working MemoryObj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

SR11

y-aligned z-aligned +xright-of

Prep. Learning

Word – Category Mapping

19

Refining learned knowledge

Is this triangle also to the right of the square?

Disjunctive primitive relationships in training are removed.

20

Learning is interactive

• Learning can be initiated by instructor or agent

“What is behind the blue sphere?”- “I don’t know the preposition behind. Please teach me with

examples.”“The green square is behind the red triangle.” “Finished teaching.”- “The blue square is behind the blue sphere.”

21

Spatial Projection

“Put the object to the right of the pantry.”

Z

Y

X

“right of”

22

Current Uses

• Describing the scene• Specifying objects in the scene• Tracking objects and relationships– “The goal is that the red triangle is inside the

pantry”• Projecting relationships back to a point in 3D

space to perform actions

23

Evaluation

Nuggets• Learns very fast• Works well for simple relationships and projection/tracking

Coals• Cannot learn complex relationships with many objects• Cannot learn certain types of relationships such as diagonal• Not robust to mistakes in training examples

24

Future Work

• Complex multi-object relationships– For example in a line or a circle

• More complex spatial relationships– Using distance/contacting etc.– Pattern matching (diagonal)– Will require more training examples

• Projection with multiple relationships– “put this inside the pantry and to the right of the blue object”

25

Questions?