David L. Chen

Fast Online Lexicon Learning for Grounded Language Acquisition

The 50th Annual Meeting of the Association for Computational Linguistics (ACL)

July 9, 2012

The University of Texas at Austin Google

Navigation Task

• Learn to interpret and follow free-form navigation instructions – e.g. Go down this hall and make a right when you see an

elevator to your left • Learn by observing how humans follow instructions• Assume no prior linguistic knowledge• Use virtual worlds and instructor/follower data

from MacMahon et al. (2006)

Sample Instructions•Take your first left. Go all the way down until you hit a dead end.

• Go towards the coat hanger and turn left at it. Go straight down the hallway and the dead end is position 4.

•Walk to the hat rack. Turn left. The carpet should have green octagons. Go to the end of this alley. This is p-4.

•Walk forward once. Turn left. Walk forward twice.

Start 3

H 4End

Sample Instructions

3

H 4

•Take your first left. Go all the way down until you hit a dead end.

• Go towards the coat hanger and turn left at it. Go straight down the hallway and the dead end is position 4.

•Walk to the hat rack. Turn left. The carpet should have green octagons. Go to the end of this alley. This is p-4.

•Walk forward once. Turn left. Walk forward twice.Observed primitive actions:

Forward, Left, Forward, Forward

Start

End

Overall System (Chen and Mooney 2011)

Learning system for parsing navigation instructions

Observation

Instruction

World State

Execution Module (MARCO)

Instruction

World State

TrainingTesting

Action TraceNavigation Plan Constructor

Semantic Parser Learner

Plan Refinement

Semantic Parser

Action Trace

Potential Navigation Plans

Instruction: Turn and walk to the couchAction Trace: Left, Forward, ForwardBackground knowledge: Layout of the map

Potential Navigation Plans

Instruction: Turn and walk to the couchAction Trace: Left, Forward, ForwardBackground knowledge: Layout of the map

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Plan Refinement

Turn and walk to the couch

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Plan Refinement

Face the blue hall and walk 2 steps

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Plan Refinement

Turn left. Walk forward twice.

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Plan Refinement

• Find the correct subplan that corresponds to the instruction

• First learn the meaning of words and short phrases

• Use the learned lexicon to remove parts of the plans unrelated to the instructions

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

1. As an example comes in, break down the sentence and the graph into n-grams and connected subgraphs

Verify TravelTurn Verify

LEFT2

steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Connected subgraph of size 1

Connected subgraph of size 2

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

Turn and walk to the couch

turn, and, walk, to, the, couch

1-gram

2-gram

3-gram

turn and, and walk, walk to, to the, the couch

turn and walk, and walk to, walk to the, to the couch

…

Turn LEFT Verify …

…

Turn

LEFT

VerifyTurn

Connected subgraph of size 1

Connected subgraph of size 2

Verify TravelTurn Verify

LEFT 2 steps

front:

BLUEHALL SOFA

front:

SOFA

at:

Subgraph Generation Online Lexicon Learning (SGOLL)

turn

Turn

…

Turn

LEFT

2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update.

48

22

26Turn

RIGHT

Subgraph Generation Online Lexicon Learning (SGOLL)

turn

Turn

…

Turn

LEFT

2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update.

48+1

22+1

26Turn

RIGHT

Subgraph Generation Online Lexicon Learning (SGOLL)

turn

Turn

…

Turn

LEFT

2. Increase the counts and co-occurrence count of each n-gram, connected-subgraph pair. Hash the connected-subgraphs for efficient update.

49

23

26Turn

RIGHT

Subgraph Generation Online Lexicon Learning (SGOLL)

turn

Turn

…

Turn

LEFT

3. Rank the entries by the scoring function

0.56

0.31

0.27

Turn

RIGHT

Evaluation Data Statistics

• 3 maps, 6 instructors, 1-15 followers/instruction• Hand-segmented into single sentence steps

Paragraph Single-Sentence

# Instructions 706 3236

Avg. # sentences 5.0 1.0

Avg. # words 37.6 7.8

Avg. # actions 10.4 2.1

Lexicon Building Time

Time in secondsChen and Mooney (2011) 2227.63SGOLL 157.3

End-to-end Execution

• Test how well the system can perform the overall navigation task

• Leave-one-map-out approach• Strict metric: Only successful if the final position

matches exactly• Upper baselines– Training with human annotated gold plans– Complete MARCO system [MacMahon, 2006]– Humans

End-to-end Execution

Single Sentences ParagraphsChen and Mooney (2011) 54.40% 16.18%Chen (2012) 57.28% 19.18%Gold Standard Plans 62.67% 29.59%MARCO 77.87% 55.69%Humans N/A 69.64%

Example ParseInstruction: “Place your back against the wall of the ‘T’ intersection.

Turn left. Go forward along the pink-flowered carpet hall two segments to the intersection with the brick hall. This intersection contains a hatrack. Turn left. Go forward three segments to an intersection with a bare concrete hall, passing a lamp. This is Position 5.”

Parse: Turn ( ), Verify ( back: WALL ),Turn ( LEFT ),Travel ( ),Verify ( side: BRICK HALLWAY ),Turn ( LEFT ),Travel ( steps: 3 ),Verify ( side: CONCRETE HALLWAY )

Mandarin Chinese Experiment

• Translated all the instructions from English to Chinese

• Train and test in the same way• Chinese does not include word boundaries

(spaces)• Naively segment each character• Use a trained Chinese word segmenter

[Chang, Galley & Manning, 2008]

Mandarin Chinese Experiment

Single Sentences ParagraphsSegmented by character 58.54% 16.11%Segmented by Stanford segmenter 58.70% 20.13%

Conclusion

• Presented a system that learns to interpret free-form navigation instructions by observing how humans follow instructions

• Assumes no prior linguistic knowledge Able to learn from multiple languages

• Fast online learning makes the system more scalable

• Thanks to my collaborators: Raymond J. Mooney and Lu Guo

• More details and data/code:http://www.cs.utexas.edu/~ml/clamp/navigation/

Questions?