ParserEvalua+on

ParserEvalua+on

‣ Viewaparseasasetoflabeledbrackets/cons+tuents

S(0,3)

NP(0,1)

PRP(0,1)(butstandardevalua+ondoesnotcountPOStags)

VP(1,3),VBD(1,2),NP(2,3),PRP(2,3)

S

NPVP

She saw it

VBD PRPPRP

0123

NP

ParserEvalua+onS(0,3),NP(0,1),VP(1,3),NP(2,3),PRP(0,1),VBD(1,2),PRP(2,3)

S

NP

She saw it

NN PRPPRP

0123

NP

S(0,3),NP(0,2),NP(2,3),PRP(0,1),NN(1,2),PRP(2,3)

‣ Precision:numberofcorrectpredic+ons/numberofpredic+ons =2/3

‣ Recall:numberofcorrectpredic+ons/numberofgolds =2/4

‣ F1:harmonicmeanofprecisionandrecall=(1/2*((2/4)-1+(2/3)-1))-1

=0.57(closertomin)

S

NPVP

She saw it

VBD PRPPRP

0123

NP

Results

‣ StandarddatasetforEnglish:PennTreebank(Marcusetal.,1993)

‣ “Vanilla”PCFG:~71F1

‣ BestPCFGsforEnglish:~90F1

‣ Otherlanguages:resultsvarywidelydependingonannota+on+complexityofthegrammar

‣ State-of-the-artdiscrimina+vemodels(usingunlabeleddata):95F1

GrammarPreprocessing

Binariza+on‣ Toparseefficiently,weneedourPCFGstobeatmostbinary(notCNF)

VP

VBD NP PP PP

sold thebook toher for$3

P(VP→VBDNPPPPP)=0.2

VP

VBD VP-[NPPPPP]

NP

PP

VP-[PPPP]

PP

‣ Solu+on:transformthetrees.Introduceintermediatespecialsymbolsthatrewritedeterminis+cally

P(VP→VBZPP)=0.1

…

P(VP→VBDVP-[NPPPPP])=0.2P(VP-[NPPPPP]→ NPVP-[PPPP])=1.0P(VP-[PPPP]→ PPPP)=1.0

PCFGIndependenceAssump+ons

11%9%

6%

NP PP DT NN PRP

9% 9%

21%

NP PP DT NN PRP

7%4%

23%

NP PP DT NN PRP

All NPs NPs under S NPs under VP

‣ Languageisnotcontext-free:NPsindifferentcontextsrewritedifferently

‣ [They]NPreceived[thepackageofbooks]NP

Ver+calMarkoviza+on

S^ROOT

NP^S VP^S

She saw it

VBD^VP PRP^VPPRP^NP

S

NP VP

She saw it

VBD PRPPRP

Basictree(v=1) v=2Markoviza+on

‣Whyisthisagoodidea?

AnnotatedTree

KleinandManning(2003)

‣ 75F1withbasicPCFG=>86.3F1withthishighlycustomizedPCFG(SOTAwas90F1atthe+me,butwithmorecomplexmethods)

‣ Augmentthegrammar:determinis+callytransformsymbolstobe“lesscontextfree”(binariza+onnotshownhere)

LexicalizedParsers§  Add“headwords”to

eachphrasalnode§  Syntac4cvs.seman4c

heads§  Headshipnotin(most)

treebanks§  Usuallyuseheadrules,

e.g.:§  NP:

§  TakeleFmostNP§  TakerightmostN*§  TakerightmostJJ§  Takerightchild

§  VP:§  TakeleFmostVB*§  TakeleFmostVP§  TakeleFchild

‣ Annotateeachgrammarsymbolwithits“headword”:mostimportantwordofthatcons+tuent

‣ Rulesforiden+fyingheadwords(e.g.,thelastwordofanNPbeforeapreposi+onistypicallythehead)

‣ CollinsandCharniak(late90s):~89F1withthese

DependencyParsing

LexicalizedParsing

S(ran)

NP(dog)

VP(ran)

PP(to)

NP(house)

DT(the) NN(house)TO(to)VBD(ran)DT(the) NN(dog)the housetoranthe dog

DependencyParsing

DT NNTOVBDDT NNthe housetoranthe dog

‣ Dependencysyntax:syntac+cstructureisdefinedbythesearcs‣ Head(parent,governor)connectedtodependent(child,modifier)‣ EachwordhasexactlyoneparentexceptfortheROOTsymbol,dependenciesmustformadirectedacyclicgraph

ROOT

‣ POStagssameasbefore,usuallyrunataggerfirstaspreprocessing

Whyaretheydefinedthisway?‣ Cons+tuencytests:‣ Subs+tu+onbyproform:thedogdidso[rantothehouse],he[thedog]rantothehouse

‣ Cleqing(Itwas[tothehouse]thatthedogran…)

‣ Dependency:verbistherootoftheclause,everythingelsefollowsfromthat

‣ Nono+onofaVP!

DependencyParsing

DT

NN

TO

VBD

DT

NN

the

house

to

ran

the

dog

‣ S+llano+onofhierarchy!Subtreesoqenalignwithcons+tuents

DependencyParsing

DT NNTOVBDDT NNthe housetoranthe dog

‣ Canlabeldependenciesaccordingtosyntac+cfunc+on

det

‣Majorsourceofambiguityisinthestructure,sowefocusonthatmore(labelingseparatelywithaclassifierworkspretywell)

nsubj

pobj

detprep

Dependencyvs.Cons+tuency:PPAtachment

‣ Cons+tuency:severalruleproduc+onsneedtochange

thechildrenatethecakewithaspoon

‣ Dependency:oneword(with)assignedadifferentparent

Dependencyvs.Cons+tuency:PPAtachment

‣Morepredicate-argumentfocusedviewofsyntax

‣ “What’sthemainverbofthesentence?Whatisitssubjectandobject?”—easiertoanswerunderdependencyparsing

‣ corenlp.run:spoonischildinsteadofwith.Thisisjustadifferentformalism

‣ Cons+tuency:ternaryruleNP->NPCCNP

Dependencyvs.Cons+tuency:Coordina+on

dogsinhousesandcats

‣ Dependency:firstitemisthehead

Dependencyvs.Cons+tuency:Coordina+on

dogsinhousesandcats

‣ Coordina+onisdecomposedacrossafewarcsasopposedtobeingasingleruleproduc+onasincons+tuency

‣ Canalsochooseandtobethehead‣ Inbothcases,headworddoesn’treallyrepresentthephrase—cons+tuencyrepresenta+onmakesmoresense

[dogsinhouses]andcats dogsin[housesandcats]

Shiq-ReduceParsing(seenotes)