Dependency Trees and Machine Translation

Vamshi AmbatiVamshi@cs.cmu.edu

Spring 2008 Adv MT Seminar

02 April 2008

Today

• Introduction– Dependency formalism– Syntax in Machine Translation

• Dependency Tree based Machine Translation– By projection– By synchronous modeling

• Conclusion and Future

Today

• Introduction– Dependency formalism– Syntax in Machine Translation

• Dependency Tree based Machine Translation– By projection– By synchronous modeling

• Conclusion and Future

Dependency Trees

John gave Mary an apple

Phrase Structure Trees

Dependency Trees

John:N gave:V Mary:N an:DT apple:N

S

VP

NP

Phrase Structure Trees: Labels

Dependency Trees

John gave Mary an apple

gave

gave

apple

Head Percolation:

- Usually done deterministically

- Assuming one head per phrase*

Dependency Trees

John Mary an

gave

apple

Dependency Trees

John Marygave an apple

Dependency Trees: Basics

• Child• Dependent• Modifier• Modifier

John gave

SUBJ

• Parent• Governor• Head• Modified

(optional)

• The direction of arrows can be head-child or child-head (has to be mentioned)

Dependency Trees: Basics

• Properties– Every word has a single head/parent

• Except for the root

– Completely connected tree– Acyclic

• If wi→wj then never wj→*wi 　

• Variants– Projective: Non-crossing between dependencies

• If wi ->wj , then for all k between i and j, either wk ->wi 　 or wk -

>wj holds

– Non-Projective: Allow crossings between depdenencies

Projective dependency tree

ounces

Projectiveness: all the words between here finally depend on either on “was” or “.”

Example credit: Yuji Matsumoto, NAIST, Japan

Non-projective dependency tree

Example from: R. McDonald and F. Pereira EACL, 2006.

Direction of edges: from a parent to the children

Note: Phrases thus extracted which are united by dependencies could be discontinuous

Dependency Grammar (DG) in the Grammar Formalism Timeline

• Panini (2600 years ago, India) recognised, distinguished and classified semantic, syntactic and morphological dependencies (Bharati, Natural Language Processing)

• The Arabic grammarians (1200 years ago, Iraq) recognised government and syntactic dependency structure, (The Foundations of Grammar - Owens)

• The Latin grammarians (800 years ago) recognised 'determination' and dependency structures. - Percival, "Reflections on the History of Dependency Notions“

• Lucien Tesnie`re (1930s, France) developed a relatively formal and sophisticated theory of DG grammar for use in schools

• PSG, CCG etc were around the same time in early 20th century

Source: ELLSSI 2000 Tutorial on Dependency Grammars

Dependency Trees: some phenomenon

• DG has been widely accepted as a variant of PSG, but it is not strongly equivalent– Constituents are implicit in a DepTree and can be

derived– Relations are explicit and can be labelled although

optional– No explicit non-terminal nodes, which means no

unary productions too– Can handle discontinuous phrases too

• Known problems with Coordination and Gerunds

Phrase structure vs Dependency

• Phrase structure suitable to languages with– rather fixed word order patterns– clear constituency structures– English etc

• Dependency structure suitable to languages with– greater freedom of word order– order is controlled more by pragmatic than by

syntactic factors– Slavonic (Czech, Polish) and some Romance (Italian ,

spanish etc)

Today

• Introduction– Dependency formalism– Syntax in Machine Translation

• Dependency Tree based Machine Translation– By projection– By synchronous modeling

• Conclusion and Future

Phrasal SMT discussion

• Advantages:– Do not have to compose translations unnecessarily– Local re-ordering captured in phrases– Already specific to the domain and capture context locally

• Disadvantages:– Specificity and no generalization– Discontiguous phrases not considered– Global reordering– Estimation problems (long vs short phrases)– Can not model phenomenon across phrases

• Limitations:– Phrase sizes (how much before I run into out of memory?)– Corpus Availability makes it feasible only to certain language

pairs

Syntax in MT: Many Representations

• WordLevel MT : No syntax

• SMT: Phrases / contiguous sequences

• SMT Hierarchical : Pseudo Syntax

• Syntax based SMT : Constituent

• Syntax based SMT: CCG

• Syntax based SMT: LFG

• Syntax based SMT: Dependency

Syntax in MT: Many ways of incorporation

• Pre-procesing– Reordering input– Reordered training corpus

• Translation models – Syntactically informed alignment models– Better distortion models

• Language Models– Syntactic language models – Syntax motivated models

• Post-processing– Nbest list reranking with syntactic information– Translation correction: Case marker/TAM correction– True casing etc?– Multi combinations with Syntactic backbones?

Syntax based SMT discussion• Inversion Transduction Grammar (Wu ‘96)

– Very constrained form of syntax : One non-terminal– Some expressive limitations– Not linguistically motivated– Effectively learns preferences for flip/no-flip

• Generative Tree to String (Yamada & Knight 2001)– Expressiveness (last week presentation)– No discontiguous phrases

• Multitext grammars (Melamed 2003)– Formalized, but MT work yet to be realized

• Hierarchical MT (Chang 2005)– Linguistic generalizations– Handles discontiguous phrases recursively– Estimation problems and Phrase table are increased even more– Across phrase boundary modeling

Syntax in MT and Dependency Trees

Source Target

Syntax

Source Target

Syntax Syntax

Tree and Tree

Tree and String

Se

Se Sf

Source side tree is provided

Target side is obtained by projection

Problem of Isomorphism between trees

• head-switching

• empty-dep ; extra-dep

Source side tree is provided

Target side is provided

Ideally non-isomorphic trees should be modeled too

Today

• Introduction– Dependency formalism– Syntax in Machine Translation

• Dependency Tree based Machine Translation– By projection– By synchronous modeling

• Conclusion and Future

Dependency Tree based Machine Translation

• By projection– Fox 2002 – Dekang Lin 2004– Quirk et al 2004, Quirk et al 2006, Menezes et al 2007

• By synchronous modeling– Alshawi et al 2001– Jason Eisner 2003– Fox 2005– Yuang Lin and Daniel Marcu 2004

Phrasal Cohesion and Statistical Machine Translation

Heidi Fox , EMNLP 2002

• English-French Corpus was used– En-Fr are similar

• For phrase structure trees -– Head Crossings involve head

constituent of the phrase with its modifier spans

– Modifier Crossings involve only spans of modifier constituents

• For dependency trees– Head Crossings means

crossings of spans of child with its parent

– Modifier crossings same as above

• Dependency structures show cohesive nature across translation

A Path-based Transfer modelDekang Lin 2004

• Input – Word-aligned– Source parsed

• Syntax translation model– Set of paths in source tree– Extract connected target path

• Generalization of paths to POS• Modeling

– Relative likelihood– Smoothing factor for noise

A Path-based Transfer modelDekang Lin 2004

• Decoding– Parse input and extract all paths, extract

target paths– Find a set of transfer rules

• Cover the entire source tree• Can be consistently merged

– Lexicalized rule preferred– Future work?

• Word ordering is addressed– Transfer rules from same sen: follow order

in sentence– Only one example of path: follow order in

rule– Many examples: pick relative distance from

head

• Highest probability

– Dynamic Programming• Min-set cover problem applied to trees

A Path-based Transfer modelDekang Lin 2004

• Evaluation– English-French: 1.2M– Source parsed by Minipar– 1755 test set– 5 to 15 words long sentences– Compared to Koehn’s results from 2003

paper• No Language Model or extra generation

module– Order defined by paths is linear– Some heuristics to maintain linearity

• Generalization of paths (transfer rules) quadratic vs. exponential

• Direct Correspondence Approach (DCA) is violated when translation divergences exist

• Very Naïve notion of reordering and merge conflict resolution

System BLEU

IBM4 0.2555

PBSMT 0.3149

Current 0.2612

Dependency Treelet TranslationQuirk et al ACL 2004,05,06

• Project dependencies from source to target via word alignment– One-one: project dependency to aligned words– Many-one: nothing to do, as the projected is the head– One-many : project to right most, and rest are

attached to it• Reattachment of modifiers to lowest possible

node that preserves target word order• Treelet extraction

– All subtrees on source until a particular limit, and the corresponding target fragment which is connected

– MLE for scoring

Dependency Treelet TranslationQuirk et al ACL 2004,05,06

tired

and

dogsmen

fatigues

et

chienshommes

tired men and dogs

hommes et chiens fatigues

et

chienshommes

fatigues

Treelet with missing roots

Dependency Treelet TranslationQuirk et al 2004,05,06

• Translation Model– Trained from the aligned

projected corpus– Log-linear with feature functions

• Channel Model– Treelet Prob– Lexical Prob

• Order Model– Head relative– Swap model

• Target Model– Target language model– Bigram Agreement model (opt)

Dependency Treelet TranslationQuirk et al ACL 2004,05,06

• Decoding (Step by step)– Input is a dependency analyzed source

• Challenge is that left-right may not work when starting with a Tree

– Obtain best target tree combining the models– Exhaustive search using DP

• Translate bottom up, from a given subtree (ITG)• For each head node extract all matching treelets: x_i

– For each uncovered subtrees extract all matching treelets: y_i

» Try all insertions of y_i into slots in x_i

» Ordering model ranks all the re-ordering possibilities for the modifiers

Dependency Treelet TranslationQuirk et al ACL 2004,05,06

• Decoding Optimizations– Duplicate translations check&reuse– Nbest list (only maintain top best candidates)– Early pruning before reordering (channel model)– Greedy reordering (pick best one and move on)– Variable n-best size (dynamically reduce ‘n’ with

increasing uncovered subtrees)– Determinstic pruning of treelets based on MLE

(allowing decoder to try more reorderings)• A* decoding

– Estimate the cost of an uncovered node reordering instead of computing it exactly

– Heuristics for optimistic estimates for each of the models

Dependency Treelet TranslationQuirk et al ACL 2004,05,06

• Evaluation– Eng-French– 1.5M parallel Microsoft

technical documentation– NLPWIN parsed on Eng

side– GIZA++ trained– Target LM: French side of

parallel data– Tuned on 250 sens for

MaxBLEU– Tested on 10K unseen– 1 Reference

Improvements to Treelet Translation

• Minimal Translation Units (HLT NAACL 2005)

– Bilingual n-gram channel model (Banchs et.al 2005)

• M = <m1,m2…>

• m1 = <si, tj>

– Instead of conditioning on the surface adjacent MTU, they condition on Headwordchain

• Dependency Order Templates (ACL 2007) Improve Generality in TranslationLearn un-lexicalised order templatesOnly use at runtime for restricting search space in reordering

Dependency Tree based Machine Translation

• By projection– Fox 2002 – Dekang Lin 2004– Quirk et al 2004, Quirk et al 2006, Menezes et al 2007

• By synchronous modeling– Alshawi et al 2001– Jason Eisner 2003– Yuang Lin and Daniel Marcu 2004– Fox 2005

Learning Dependency Translation Models as Collections of Finite-State Head Transducers

Alshwai et al 2001

• Head transducers variant– Middle-out string transduction vs. left-right– Can be used in a hierarchical fashion, if you consider

input/output for non-head transitions as ‘strings’ rather than ‘words’

• Dependency transduction model

Empty in/out

May not always be a

dependency model in

conventional sense

Learning Dependency Translation Models as Collections of Finite-State Head Transducers

Alshwai et al 2001

• Training: Given unaligned bitext– Compute coocurrence statistics at wordlevel– Find a hierarchical synchronous alignment driven by cost function– Construct a set of head transducers that explain the alignment– Calculate the transition weights by MLE

• Decoding– Similar to CKY or Chart Parsing, but ‘middle-out’– Given input, find the best applications of transducers– A derivation spanning entire input means it probably has found

best dependencies for source & target– Else string together most probable partial hypothesis to form a

tree– Pick the target tree with lowest score and read off the string

Learning Dependency Translation Models as Collections of Finite-State Head Transducers

Alshwai et al 2001• Evaluation

– Eng – Spanish (ATIS data – 13,966 train, 1185 test)– Eng – Jap (Speech transcribed data – 12,226 train, 3253 test)

• Discussion– Language agnostic, direction agnostic– Induced dependency tree may not be syntactically motivated, but suited to

translation– Application of transducers is done locally, and so less context information– A single transducer tries to do everything, training may have sparsity problems

Learning non-isomorphic tree mappings for MTJason Eisner 2003

• Non-Isomorphism not just due to language divergences but free translation

• A version of Tree Substitution Grammar– To learn from unaligned non-isomorphic trees– A statistical model based generalized instead of

linguistic minimalism– Expressive with empty string insertions– Formulate for both PSG and DG

• Translation model– Joint model P (Ts,Tt,A)

• Alignment• Decoding• Training

– Factorization helps: • Reconstruct all derivations for a tree by efficient

‘tree parsing’ algorithm for TSG• EM as an efficient inside-outside training on all

derivations

• Decoding– Chart Parsing to create a forest of derivations

for input tree– Maximize over probability of derivations– 1-best derivation parse is syntactic-alignment

1. Kids kiss Sam quite often

2. Lots of kids give kisses to Sam

Machine Translation Using Probabilistic Synchronous Dependency Insertion Grammars

Ding and Marcu 2005

• SDIG– Like STAG, STIG for phrase

structures– Basic units are elementary trees– Handles non-isomorphism at

sub-tree level

• Cross-lingual inconsistencies are handled if they appear within basic units– Crossing-dependency– Broken-dependency

Machine Translation Using Probabilistic Synchronous Dependency Insertion Grammars

Ding and Marcu 2005

• Induction of SDIG for MT as Synchronous hierarchical tree partitioning– Train IBM Mode 1 scores for

bitext– For each category of Node,

starting with NP - – Perform synchronous tree

partitioning operations • Compute Prob of word pair

(ei,fi) where operation can be performed

• Heuristic functions (Graphical model) guide the partitioning

Machine Translation Using Probabilistic Synchronous Dependency Insertion Grammars

Ding and Marcu 2005

• Translation • Decoding for MT

– Translation is obtained by • maximizing over all possible

derivations of the source tree

• translation of the ‘elementary trees’

– Analogous to HMM (Emission and Transition probs with elementary trees)

– Decoding is similar to a Viterbi-style algorithm on the tree

• Hooks– Augmenting corpus by

singleton ETs from Model1– Smoothing probabilities

Machine Translation Using Probabilistic Synchronous Dependency Insertion Grammars

Ding and Marcu 2005

• Evaluation– Chinese-English system– Dan Bikels parses for both Cn,En trained from Parallel

treebanks– Test with 4 refs

• Compared with– GIZA trained– ISI Rewrite Decoder

• NIST increased 97%• BLUE increased 27%• Reordering ignored for now

Dependency Based Statistical MT Fox 2005

• Czech-English parallel corpus (Penn TB and Prague TB)– Morphological process and tecto-grammatical

conversion for Czech trees– No processing for English trees

• Alignment of subtrees via IBM Model4 scores – followed by structural modification of trees to suit

alignment (KEEP,SPLIT,BUD…)

• Translation Model :

Dependency Based Statistical MT Fox 2005

• Decoding– Bestfirst decoder– Process given Czech input to dependency tree and translate each

node independently– For each node

• Choose head position• Generate english POS seq• Generate the feature list• Perform structural mutations

• Syntax Language Model– Takes as input a forest of phrase structures– Invert decoder forest output (dep tree nodes) into phrase structures– Reordering is entirely left to LM

• Evaluation– Work in progress – Proposed to use BLEU

Today

• Introduction– Dependency formalism– Syntax in Machine Translation

• Dependency Tree based Machine Translation– By projection– By synchronous modeling

• Conclusion and Future

Conclusion• The good -

– Easy to work with– Cohesive during projection– Builds well on top of existing PBSMT (Effective combination

of lexicalization and syntax)– Supports modeling a target even with crossing phrase

boundaries– Gracefully degrade over new domains

• The bad –– Reordering is not crucial, but expensive – Lots of hooks for decoding– Generalization explodes space

• The not so good –– Current approaches require a dependency tree on source

side and a strong model for the target side

What Next…• 1 year

– Better scoring and estimation in syntactic translation models– Improvement in Dependency trees parse quality directly translates?

(Chris Quirk et al 2006) ? What about MST Parser etc?– Better Word-Alignment and effect on model– Incorporating labeled dependencies. Will it help? – Factored Dependency Tree based models– Approximate subtree matching and Parsing algorithms

• 3-5 years– Decoding Algorithms and the Target-Ordering problem– Discriminative approaches to MT are catching up. How can syntax be

incorporated into such a framework – Better syntactic language models based on the dependency

formalisms– Semantics in Translation (Are DepTrees the first step?)– Fusion of Dependency and Constituent approaches (LFG style)– Joint Modeling approaches (Eisner 03, Smith 06 QS Grammar)– Taking MT to other applications like Cross-lingual Retrieval and QA

which already use DepFormalisms

Thanks to

• Lori Levin: For discussion on Dependency tree formalism

• Amr Ahmed: For discussion and separation of work

• Respective authors of the papers for some of the graphic images I liberally used in the slides

Questions

Thanks

DG Variants

• Case Grammar (Anderson)• Daughter-Dependency Theory (Hudson)• Dependency Unification Grammar (Hellwig)• Functional-Generative Description (Sgall)• Lexicase (Starosta)• Meaning-Text Model (Mel'cuk)• Metataxis (Schubert)• Unification Dependency Grammar (Maxwell)• Constraint Dependency Grammar (Maruyama)

Motivation Questions

• 1. How is dependency analysis used in Syntax MT? How do the algorithms vary if only the source side of analysis is present?

• 2. How do the decoding and transfer phases adapt when using dependency analysis? What algorithms exist and what is the complexity analysis?

• 3. How does dependency based syntax incorporation in MT, compare with other grammar formalisms like the phrase structure grammar?

• 4. Is there a class of languages which yield better to dependency analysis vs. other analysis?

• 5. Dependency analysis being close to semantics, does it help MT produce better results?

Other Papers

• QuasiSynchronous Grammars for Soft Syntactic Projection David Smith and Jason Eisner 2007

• Automatic Learning of Parallel Dependency Treelet Pairs Yuan Ding and Martha Palmer 2004

• Dependency vs. Constituents for Tree-Based Alignment Dan Gildea 2003

• My Compilation:– http://kathmandu.lti.cs.cmu.edu:8080/wiki/index.php/AMT:Sched

ule