Towards a Universal Grammar for Natural Language Processing · Towards a Universal Grammar for...

transcript

Towards a Universal Grammar for Natural Language Processing

Joakim Nivre

Uppsala UniversityDepartment of Linguistics and Philology

Based on collaborative work with Filip Ginter, Yoav Goldberg, Jan Hajič, Chris Manning, Ryan McDonald, Natalia Silveira, Marie de Marneffe, Slav Petrov,

Sampo Pyysalo, Reut Tsarfaty, Daniel Zeman and many others

“In its substance, grammar is one and the same in all languages, even if it accidentally varies.”

Universal Grammar

• All human languages are species of a common genus

Universal Grammar

• Language structure is constrained by a universal cause

Universal Grammar

• Language structure is constrained by a universal cause

• There is order in the chaos of linguistic variation

Natural Language Processing

• Linguistic diversity makes our life harderWhy 90% parsing accuracy for English but only 80% for Finnish?

Can we even compare the numbers?

• Current NLP relies heavily on linguistic annotation:“In its substance, grammar is the same in all languages, even if the annotation accidentally varies.”

• We need to bring some order into the chaos

En katt jagar rattor och moss

det nsubj conj

En kat jager rader og møs

? dobj cc conj

A cat chases rats and mice

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language Y

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language YEn katt jagar rattor och moss

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language Z

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language Z

Which languages are most closely related?

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language Z

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language Z

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language Z

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language X

det nsubj conj

? dobj cc conj

det nsubj dobj cc

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Language Z

Swedish

Danish

English

det nsubj conj

En kat jager rotter og mus

? dobj cc conj

det nsubj dobj cc

Toutefois , les filles adorent les desserts .

toutefois , les fille adorer les dessert .

ADV PUNCT DET NOUN VERB DET NOUN PUNCT

Definite=Def Gender=Fem Number=Plur Definite=Def Gender=MascNumber=Plur Number=Plur Person=3 Number=Plur Number=Plur

Tense=Pres

advmod

det nsubj

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

Why is this a problem?

• Hard to compare empirical results across languages

• Hard to evaluate cross-lingual learning

• Hard to build and maintain multilingual systems

• Hard to make progress towards a universal parser

Universal Dependencieshttp://universaldependencies.org

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

Part-of-speech tags

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

Part-of-speech tags

Morphological features

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

Part-of-speech tags

Morphological features

Dependency relations

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

Stanford Dependencies

Google UD

Stanford UD

Google UD

Stanford UD

HamleDT

Google UD

Stanford UD

HamleDT

Interset

Google UD

Stanford UD

HamleDT

Interset

Google universal tags

Universal Dependencies

• Milestones:• Kick-off meeting at EACL in Gothenburg, April 2014• Release of annotation guidelines, Version 1, October 2014• Release of treebanks for 10 languages, January 2015• Release of treebanks for 18 languages, May 2015• Release of treebanks for 33 languages, November 2015

• Open community effort – anyone can contribute!

Goals and Requirements

• Cross-linguistically consistent grammatical annotation

• Support multilingual research and development in NLP

• Based on common usage and existing de facto standards

Design Principles

• Dependency• Widely used in practical NLP systems

• Available in treebanks for many languages

Design Principles

• Dependency• Widely used in practical NLP systems

• Available in treebanks for many languages

• Lexicalism• Basic annotation units are words – syntactic words

• Words have morphological properties

• Words enter into syntactic relations

Golden Rules

Maximize parallelism• Don’t annotate the same thing in different ways

• Don’t make different things look the same

Golden Rules

Maximize parallelism• Don’t annotate the same thing in different ways

• Don’t make different things look the same

But don’t overdo it• Don’t annotate things that are not there

• Languages select from a universal pool of categories

• Allow language-specific extensions

Morphology

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

• Lemma representing the semantic content of the word

det nsubj conj

? dobj cc conj

det nsubj dobj cc

toutefois , le fille adorer le dessert .

Tense=Pres

advmod

det nsubj

toutefois , le fille adorer les dessert .

Tense=Pres

advmod

det nsubj

Morphology

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

• Part-of-speech tag representing the abstract lexical category associated with the word

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

Tense=Pres

advmod

det nsubj

Morphology

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

• Part-of-speech tag representing the abstract lexical category associated with the word

• Features representing lexical and grammatical properties associated with the lemma or the particular word form

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

Tense=Pres

advmod

det nsubj

Morphology

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

det nsubj conj

? dobj cc conj

det nsubj dobj cc

Tense=Pres

advmod

det nsubj

• Taxonomy of 17 universal part-of-speech tags, based on the Google Universal Tagset (Petrov et al., 2012)

• All languages use the same inventory, but not all tags have to be used by all languages

Part-of-Speech TagsOpen Closed OtherADJ ADP PUNCT

ADV AUX SYM

INTJ CONJ X

NOUN DET

PROPN NUM

VERB PART

Features

• Standardized inventory of morphological features, based on the Interset system (Zeman, 2008)

• Languages select relevant features and can add language-specific features or values with documentation

LexicalInflectional Nominal

Inflectional Verbal

PronType Gender VerbForm

NumType Animacy Mood

Poss Number Tense

Reflex Case Aspect

Definite Voice

Degree Person

Negative

Syntax

The cat could have chased all the dogs down the street .

DET NOUN AUX AUX VERB DET DET NOUN ADP DET NOUN PUNCT

Syntax

• Content words are related by dependency relations

Syntax

• Function words attach to the content word they modify

Syntax

• Function words attach to the content word they modify

• Punctuation attach to head of phrase or clause

The dog was chased by the cat .DET NOUN AUX VERB ADP DET NOUN PUNCT

nsubjpass

auxpass

Hunden jagades av katten .NOUN VERB ADP NOUN PUNCT

Definite=Def Voice=Pass Definite=Def

nsubjpass

auxpass

nsubjpass

auxpass

nsubjpass

auxpass

nsubjpass

Dependency Relations

• Taxonomy of 40 universal grammatical relations, broadly attested in language typology (de Marneffe et al., 2014)• Language-specific subtypes may be added

Dependency Relations

• Taxonomy of 40 universal grammatical relations, broadly attested in language typology (de Marneffe et al., 2014)• Language-specific subtypes may be added

• Organizing principles• Three types of structures: nominals, clauses, modifiers

• Core arguments vs. other dependents (not complements vs. adjuncts)

Dependents of Clausal Predicates

Nominal Clausal Other

nsubjnsubjpass

dobjiobj

csubjcsubjpassccompxcomp

Non-Core

nmodvocative

discourseexpl

advmodnegaux

auxpasscop

markpunct

Mary was quietly reading a book in the garden .PROPN AUX ADV VERB DET NOUN ADP DET NOUN PUNCT

advmod

dobj case

If you are sick , you should not exercise .SCONJ PRON AUX ADJ PUNCT PRON AUX ADV VERB PUNCT

Peter thought that he should stop smoking .PROPN VERB SCONJ PRON AUX VERB VERB PUNCT

Cairo , the lovely capital of EgyptPROPN PUNCT DET ADJ NOUN ADP PROPN

Hewey , Dewey and LouiePROPN PUNCT PROPN CONJ PROPN

advmod

dobj case

advmod

dobj case

advmod

dobj case

advmod

dobj case

advmod

dobj case

advmod

dobj case

Helsinki , the lovely capital of FinlandPROPN PUNCT DET ADJ NOUN ADP PROPN

Huey , Dewey and LouiePROPN PUNCT PROPN CONJ PROPN

Dependents of Nominals

Nominal Clausal Other

nummodapposnmod

aclamoddetcase

• Coordinate structures are headed by the first conjunct• Subsequent conjuncts depend on it via the conj relation

• Conjunctions depend on it via the cc relation

• Punctuation marks depend on it via the punct relation

advmod

dobj case

Huey , Dewey and LouiePROPN PUNCT PROPN CONJ PROPN

CoordinationCoordination

(punct)

Multiword Expressions

• UD annotation does not permit “words with spaces”• Multiword expressions are analysed using special relations

• The mwe, name and goeswith relations are always head-initial

• The compound relation reflects the internal structure

Relation Examples

mwe in spite of, as well as, ad hoc

name Roger Bacon, Carl XVI Gustaf, New York

compound phone book, four thousand, dress up

goeswith notwith standing, with out

Other Relations

Relation Explanation

parataxis Loosely linked clauses of same rank

list Lists without syntactic structure

remnant Orphans in ellipsis linked to parallel elements

reparandum Disfluency linked to (speech) repair

foreign Elements within opaque stretches of code switching

dep Unspecified dependency

root Syntactically independent element of clause/phrase

Language-Specific Relations• Language-specific relations are subtypes of universal

relations added to capture important phenomena

• Subtyping permits us to “back off” to universal relations

Relation Explanation

acl:relcl Relative clause

compound:prt Verb particle (dress up)

nmod:poss Genitive nominal (Mary’s book)

nmod:agent Agent in passive (saved by the bell)

cc:preconj Preconjunction (both … and)

det:predet Predeterminer (all those …)

Where are we now?

Where are we now?• Universal Dependencies, Version 1

• Guidelines released October 2014

• Latest treebank release November 2015 (v1.2):Ancient Greek, Arabic, Basque, Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Gothic, Greek, Hebrew, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Latin, Norwegian, Old Church Slavonic, Persian, Polish, Portuguese, Romanian, Slovenian, Spanish, Swedish, Tamil

• Future plans:• New releases every six months (May, November)

• Revision of guidelines as needed

• Future plans:• New releases every six months (May, November)

• Revision of guidelines as needed

• Have a look at http://universaldependencies.org

So what exactly is UD?

So what exactly is UD?• A new linguistic theory?

Not at all, but we like to think it is informed by linguistic theory and potentially useful also for linguistic studies

• A better parsing framework?Probably not, since parsers seem to prefer function words as heads so we may have to tweak the representations for parsing

• The ultimate annotation scheme?Not quite, more like a lingua franca for treebank developers and definitely useful for some annotation projects

• A universal grammar?Not in the Chomskyan sense, but hopefully in the more practical sense of facilitating multilingual NLP by bringing a little order into the chaos

Well, who knows?

Acknowledgments• Core UD group: Filip Ginter, Yoav Goldberg, Jan Hajič, Chris Manning, Ryan

McDonald, Natalia Silveira, Marie de Marneffe, Slav Petrov, Sampo Pyysalo, Reut Tsarfaty, Dan Zeman

• UD contributors: Željko Agić, Riyaz Ahmad, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Cristina Bosco, Giuseppe G. A. Celano, Jinho Choi, Çağrı Çöltekin, Kaja Dobrovoljc, Timothy Dozat, Binyam Ephrem, Tomaž Erjavec, Richárd Farkas, Jennifer Foster, Koldo Gojenola, Iakes Goenaga, Bruno Guillaume, Nizar Habash, Dag Haug, Anders Trærup Johannsen, Hiroshi Kanayama, Jenna Kanerva, Simon Krek, Juha Kuokkala, Veronika Laippala, Alessandro Lenci, Krister Lindén, Nikola Ljubešić, Olga Lyashevskaya, Teresa Lynn, Aibek Makazhanov, Catalina Maranduc, Héctor Martínez Alonso, Anna Missilä, Verginica Mititelu, Yusuke Miyao, Simonetta Montemagni, Shinsuke Mori, Hanna Nurmi, Petya Osenova, Lilja Øvrelid, Elena Pascual, Marco Passarotti, Jussi Piitulainen, Barbara Plank, Prokopis Prokopidis, Loganathan Ramasamy, Wolfgang Seeker, Mojgan Seraji, Maria Simi, Kiril Simov, Arne Skjæerholt, Aaron Smith, Jan Štěpánek,Takaaki Tanaka, Francis Tyers, Sumire Uematsu, Veronika Vincze, Rob Voigt, Jonathan Washington

Towards a Universal Grammar for Natural Language Processing · Towards a Universal Grammar for...

Documents