New  Algorithms  for  Seman3cs-­‐Based  Machine  Transla3on

USC/ISI

2012  November  2

Why  Seman3cs?

• Explicit  meaning  representa0on  will  improve  meaning-­‐preserving  transla0on

• Modeling  meaning  instead  of  surface  realiza0ons  →  learn  more  from  less  data

2

sourcestring

targetstring

NLU NLG

WANT

BOY

GO

instance

instance

instanceARG0

ARG1

ARG0

Rooted,  edge-­‐labeled,leaf-­‐labeled  graph

The  boy  wants  to  go. 男孩子想去。

∃w,  b,  g  :  instance(w,  WANT)  ^                            instance(g,  GO)  ^                            instance(b,  BOY)  ^                            agent(w,  b)  ^                              pa0ent(w,  g)  ^                            agent(g,  b)  

Five  Equivalent  Meaning  Representa3on  Formats

(w  /  WANT    :agent  (b  /  BOY)    :pa0ent  (g  /  GO                                          :agent  b)))

WANT

BOY

GO

instance

instance

instanceagent

pa0ent

agent

((x0  instance)  =  WANT((x1  instance)  =  BOY((x2  instance)  =  GO((x0  agent)  =  x1((x0  patent)  =  x2((x2  agent)  =  x1

instance:    WANTagent:pa0ent: instance:  GO

agent:

instance:  BOY1

1

LOGICAL  FORM PATH  EQUATIONS

FEATURE  STRUCTURE

DIRECTED  ACYCLIC  GRAPH

PENMAN

“The  boy  wantsto  go.”

Mul3ple  Roles   Graph  Structure

“Pascale  was  charged  with    public  intoxica3on  and    resis3ng  arrest.”

instance

charge-­‐05

ARG1

ARG2

instance name

person Pascale

...

Mul3ple  Roles   Graph  Structure

“Pascale  was  charged  with    public  intoxica3on  and    resis3ng  arrest.”

instance

charge-­‐05

ARG1

ARG2

instance name

person Pascale

op2

op1

instanceand

instance

intoxicate-­‐01 resist-­‐01

instance

Mul3ple  Roles   Graph  Structure

“Pascale  was  charged  with    public  intoxica3on  and    resis3ng  arrest.”

instance

charge-­‐05

ARG1

ARG2

op2

op1

instanceand

instance

intoxicate-­‐01

public

instance

loca3on

resist-­‐01

instance name

instance

person Pascale

Mul3ple  Roles   Graph  Structure

“Pascale  was  charged  with    public  intoxica3on  and    resis3ng  arrest.”

instance

charge-­‐05

ARG1

ARG2

op2

op1

instanceand

instance

intoxicate-­‐01

public

instance

loca3on

resist-­‐01ARG1

instance name

instance

person Pascale

Mul3ple  Roles   Graph  Structure

“Pascale  was  charged  with    public  intoxica3on  and    resis3ng  arrest.”

instance

charge-­‐05

ARG1

ARG2

op2

op1

instanceand

instance

intoxicate-­‐01

public

instance

loca3on

resist-­‐01ARG1

ARG0

instance name

instance

person Pascale

Mul3ple  Roles   Graph  Structure

“Pascale  was  charged  with    public  intoxica3on  and    resis3ng  arrest.”

instance

charge-­‐05

ARG1

ARG2

op2

op1

instanceand

instance

intoxicate-­‐01

public

instance

loca3on

resist-­‐01ARG1

ARG0

arrest-­‐01

instance

instance name

ARG1instance

person Pascale

Mul3ple  Roles   Graph  Structure

“Pascale  was  charged  with    public  intoxica3on  and    resis3ng  arrest.”

instance

charge-­‐05

ARG1

ARG2

op2

op1

instanceand

instance

intoxicate-­‐01

public

instance

loca3on

resist-­‐01ARG1

ARG1ARG0

arrest-­‐01

instance

instance name

ARG1instance

person Pascale

Meaning-­‐based  MT

• Too  big  for  just  this  MURI:–What  content  goes  into  the  meaning  representa3on?• linguis3cs,  annota3on  (Nathan  Schneider,  CMU)

–How  are  meaning  representa3ons  probabilis3cally  generated,  transformed,  scored,  ranked?• automata  theory,  efficient  algorithms

–How  can  a  full  MT  system  be  built  and  tested?• engineering,  language  modeling,  features,  training

Abstract  Meaning  Representa0on•    35-­‐page  guidelines.•    Extensive  use  of  PropBank  predicates,        but  cover  all  words  in  sentence.•    AMR  Editor  with  logins  and  worksets.•    7  minutes  per  sentence,  enabling  very        large-­‐scale  Sembanking.

• Too  big  for  just  this  MURI:–What  content  goes  into  the  meaning  representa3on?• linguis0cs,  annota0on

–How  are  meaning  representa3ons  probabilis3cally  generated,  transformed,  scored,  ranked?• automata  theory,  efficient  algorithms

–How  can  a  full  MT  system  be  built  and  tested?• engineering,  language  modeling,  features,  training

Meaning-­‐based  MT

General-­‐Purpose  Algorithms  for  Feature  Structures  (Graphs)

String  World

(words)

Tree  World

(syntax)

Graph  World?

(seman0cs)Acceptor Finite-­‐state  acceptors Tree  automata

Transducer Finite-­‐state  transducers Tree  transducers

Membership  checking O(n) O(n)  for  trees

O(n3)  for  strings

N-­‐best  …   …  paths  through  an  WFSA  

(Viterbi,  1967;  Eppstein,  1998)

…  trees  in  a  weighted  forest    (Jiménez  &  Marzal,  2000;    Huang  &  Chiang,  2005)

EM  training Forward-­‐backward  EM                    (Baum/Welch,  1971;  Eisner  2002)

Tree  transducer  EM  training  

(Graehl  &  Knight,  2004)

Intersec;on WFSA  intersec0on Tree  acceptor  intersec0on  

Transducer  composi;on WFST  composi0on                            

(Pereira  &  Riley,  1996)

Many  tree  transducers  not  closed  under  composi0on                      

(Malep  et  al  09)

General  tools Carmel,  OpenFST Tiburon  (May  &  Knight  10)

Hyperedge  Replacement  Grammars

–Survey:  Drewes  et  al.,  1997

–Several  NLP-­‐related  publica3ons  from  USC/ISI  forthcoming

–Key  idea:  context-­‐free  rewri3ng

16

HRG  Deriva3on

instance

ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

=  boy  wants  girl  to  believe  that  he  is  wanted

LET’S  DERIVE  THIS:

HRG  Deriva3on

instance

ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

“the  boy  wantssomethinginvolving  himself”

LET’S  DERIVE  THIS:

=  boy  wants  girl  to  believe  that  he  is  wanted

instance

WANT

B

X

ARG0

ARG1

HRG  Deriva3on

instance

ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

“the  boy  wantssomethinginvolving  himself”

instance

WANT

B

LET’S  DERIVE  THIS:

=  boy  wants  girl  to  believe  that  he  is  wanted

ARG0

ARG1

X

HRG  Deriva3on

instance

ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

instanceARG0

WANT

B

Xinstance

ARG0

BELIEVE

G

“the  boy  wants  the  girl  to  believesomething  involving  him”

ARG1

LET’S  DERIVE  THIS:

=  boy  wants  girl  to  believe  that  he  is  wanted

HRG  Deriva3on

instance

ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

instanceARG0

WANT

B

Xinstance

ARG0

BELIEVE

G

“something  involving  B”

ARG1

LET’S  DERIVE  THIS:

=  boy  wants  girl  to  believe  that  he  is  wanted

HRG  Deriva3on

instance

ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

instanceARG0

WANT

B

instance

ARG0

BELIEVE

G

instance

WANT

ARG1

ARG1

ARG1

FINISHED!

LET’S  DERIVE  THIS:

=  boy  wants  girl  to  believe  that  he  is  wanted

General-­‐Purpose  Algorithms  for  Feature  Structures  (Graphs)

String  World

(words)

Tree  World

(syntax)

Graph  World

(seman0cs)Acceptor Finite-­‐state  acceptors Tree  automata HRG

Transducer Finite-­‐state  transducers Tree  transducers Synchronous  HRG

Membership  checking O(n) O(n)  for  trees

O(n3)  for  strings

O(nk+1)  for  graphs

N-­‐best  …   …  paths  through  an  WFSA  

(Viterbi,  1967;  Eppstein,  1998)

…  trees  in  a  weighted  forest    (Jiménez  &  Marzal,  2000;    Huang  &  Chiang,  2005)

…  graphs  in  a  weighted  forest

EM  training Forward-­‐backward  EM                    (Baum/Welch,  1971;  Eisner  2003)

Tree  transducer  EM  training  

(Graehl  &  Knight,  2004)

EM  on  forests  of  graphs

Intersec;on WFSA  intersec0on Tree  acceptor  intersec0on   Not  closed

Transducer  composi;on WFST  composi0on                            

(Pereira  &  Riley,  1996)

Many  tree  transducers  not  closed  under  composi0on                      

(Malep  et  al  09)

Not  closed

General  tools Carmel,  OpenFST Tiburon  (May  &  Knight  10) Bolinas

SHRG  Deriva3on

S

wantsB INF

instanceARG0

WANT

B

X

“the  boy  wantssomethinginvolving  himself”

SHRG  Deriva3on

S

wantsB INF

to  believeG S

instanceARG0

WANT

B

X

ARG1

instance

ARG0

BELIEVE

G

“something  involving  B”

SHRG  Deriva3on

instanceARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

G

instance

WANT

ARG1

S

wantsB INF

to  believeG S

is  wantedhe

FINISHED!

ARG1

sourcestring

targetstring

NLU NLG

WANT

BOY

GO

instance

instance

instanceARG0

ARG1

ARG0

Rooted,  edge-­‐labeled,leaf-­‐labeled  graph

The  boy  wants  to  go. 男孩子想去。

SHRGSHRG

• Task: Given a graph H, find the best

• derivation of H

• transduction of H into a syntactic tree

Recognizing (hyper)graphs

arg1

wantʹ

believeʹ

boyʹ

girlʹ

arg1S

NP VP

the boy V S

NP VPwants

the girl V

to believe

NP

him

arg0

Recognizing (hyper)graphs

• Previous algorithms (Drewes 1997) fairly theoretical

• What we want:

• Extract a bunch of HRG rules

• Throw out the troublemakers

• Guarantee recognition in O(nk) time

Pathwidth

Pathwidth

Pathwidth

Pathwidth

Pathwidth

Treewidth

• Pathwidth k = a single boundary with at most k nodes

• Treewidth k = multiple independent boundaries, each with at most k nodes

Recognizing (hyper)graphs

• If rules have treewidth at most k, we can recognize in time O(nk+1)

want′ arg1

X

E

arg0

treewidth 2

Recognizing (hyper)graphs

treewidth runtime continents strings(words)

trees(syntax)

graphs(semantics)

1 n2 CFG

HRG in practice

2 n3 AustraliaS America

CFG TAGHRG in practice

3 n4 Africa

HRG in practice

4 n5 N AmericaEurope

HRG in practice

5 n6 Asia TAG